New intro

Introduction To ARROW

A few years ago, TRACC combined the original hardware from the Phoenix and Zephyr clusters into the ARROW cluster. This consolidation allowed efficient administration of TRACC cluster services with limited staff. To avoid the problems of load balancing, the different types of hardware nodes on the ARROW cluster were partitioned and available in queues. As new hardware was installed to expand cluster resources, it was made available via new queues. ARROW is arranged such that there is a single set of 4 login nodes, a singe file system, and single user home directory that serves all of the nodes in all of the queues. Jobs were scheduled on Arrow using Torque and Maui as the scheduler.

We have been using Torque and Maui as our queuing system. Jobs can be submitted from any of the login nodes. Once a job starts, the nodes assigned to that user are generally accessible by additional ssh sessions from any other node in the system. For example, if you submit a job from login1, you can go to login2 and create an ssh session to the node that was handed out by the scheduler. Think of it as a global resource allocation that gives you access to a few nodes that you can do anything on as you desire until the job time expires. This is true for interactive and batch sessions, it’s all the same. Any node assigned to a user is fully allocated to that user, and a job can only ask for full nodes. No other users can share a node that has been handed out to a user. The queues are used to get certain CPU types for the job.

Recently OpenPBS (or PBS-PRO), a new scheduler has been installed but not activated by default. It behaves roughly like the old scheduler, but is much more modern and flexible. Because it is installed but not activated, you have to turn off the standard scheduler and enable OpenPBS. Our method of switching is using the “module” command. The “module” command changes your environment variable to enable or disable certain software collections or components. That includes Python, OpemFoam, StarCCM+, and many more. This is all configured for the current production system. The module command is:

 
 module load pbs/pbs

After that command has been issued in a specific terminal window, everything you do in that exact terminal window will be using the new scheduler. If you want to have that same capability from another terminal window, issue the “load” command again. Now that second terminal operates with OpenPBS as well. So you can login to any of the login servers and use the above “load” command to switch to OpenPBS. We are operating all nodes on the cluster either under the old scheduler or the other one. As an example, we have added some Therefore, you can’t see the GPU nodes from the old scheduler any more.

The syntax of the scheduler commands is a bit different allowing for more concise job specifications. By default, users can ask for specific queues, number of nodes, number of cores per node, number of gpus, and amount of memory. If space is available on a node already partially in use by somebody else, your job may start sharing the same node. Things like the wall time of concurrent jobs are independent, so think of each job to be executed within the resources specified at submission time. It’s not very different from the current system, but much more fine grained.

The operating system uses CGROUPS to keep resources from affecting each other. If your job starts using more cores than you asked for, the operating system will pack all your load onto the allocated cores. Therefore, your job cannot eat into the resources if other jobs tuning in the same machine. If your job starts to use more memory than allocated to you, your job dies. There may be a few ways users can mess with other user’s jobs, bit they would have to be hacking things. The environment modules software switches as well when using the new scheduler.

Important- Not all software is available yet to fully use the functions of the PBS scheduler. New nodes will be added that will be accessible be the PBS scheduler, and existing nodes may be moved from being accessible by Torque to being accessible by PBS. Initially the PBS accessible nodes will be mostly usied testing compatibility right now. The documentation at Using the Cluster describes procedures for using ARROW.