TRACC Wiki - User contributions [en]

Setting Up Your Environment

2025-04-16T22:54:39Z

Amiot: /* Changing ARROW Computing Queues */

=== Shells ===

* Your account will be set up with the bash shell, unless you asked for a different shell when you requested your account. You can find out your current shell with the command <code>echo $SHELL</code>.

* If you'd like a different shell, please contact us. [mailto:TRACC_Director@anl.gov Click to send email].

* The commands <code>chsh</code> and <code>ypchsh</code> don't work on the cluster, and have been disabled.

=== Accessing application software with modules ===

* The cluster uses the [http://modules.sourceforge.net Environment Modules] package to manage access to application software.

* modules documentation is available with the command <code>man module</code>.

* To set up your environment so that an application can be used, use the command

module load <module>

at the command line or in your shell's configuration files.

* You can undo the effect of a module load with

module unload <module>

* It's generally not a good idea to load modules for more than one version of the same application. Instead, swap the modules:

module swap some-application/stable some-application/beta

* Your .bashrc or .cshrc contains sample commands for loading the most common modules. Please see the instructions in those files, or in /etc/skel/.bashrc or /etc/skel/.cshrc, which will be the latest versions of the default configuration files.

* Most modules are one of these two forms:

<application type>/<application>/<version>
<application>/<version>

for example

compiler/pathscale/3.1-147
starccm/15.06.008-R8

Omitting the version number will load the latest stable version of the module.

To see what modules are currently loaded in your environment, do

module list

To see what modules are available, do

module avail

* Modules with "-alpha" or "-beta" appended to the name may not work, either because the software or module is broken.

* To use modules in shell scripts, including Torque jobs scripts, see [[Running Your Code#Job_scripts| below]].

=== Changing ARROW Computing Queues ===
There are currently several computing queues that are available to schedule and some of these queues can only be scheduled by one scheduler (Torque) and the others by a second scheduler (PBS). The nodes scheduled by the queues vary from queue to queue. Be aware that node characteristics (RAM, number of processors, etc) vary from queue to queue. The Torque scheduler is available by default; the PBS scheduler is only available by loading its environmental variable:

The queues "PBS Computing Queues" are available after the environmental specifying the PBS Scheduler is initiated

<pre>
module load pbs/pbs
</pre>
You can review the queues that are available and which scheduler to use to [[ARROW Cluster |'''schedule them by clicking here.''']]

=== File Permissions ===

By default, you will be in your own Unix group, and your files will be readable and writable only by you (that is, the default umask is 027). If you want files to be readable by everyone working on your project, you will need to

* Have your project leader request that we create a Unix group for the project. They also need to let us know who should be in the group.

* Change the group ownership of the files you want to be group-readable:

chgrp <project-group> <filename>

* Make the file group-readable, and, if user-executable, group-executable:

chmod g+rX <filename>

If you want all your new files to be readable by everyone in your group, you'll need to

* Ask us to change your default group. To contact us
[mailto:TRACC_Director@anl.gov click to send email].

Note that a change in your primary group will not affect pre-existing files. To change the group ownership of all your files, do this:

chgrp -R <new-group> ~
chmod -R g+rX ~
chgrp -R $USER ~/.ssh
chmod -R g-rx ~/.ssh

The last two commands are necessary for SSH security; you will lock yourself out of the cluster if you omit them.

Running Your Code

2025-04-16T22:14:23Z

Amiot: /* Getting started */

= Careful, this chapter is quite outdated and needs to be rewritten =

== Running Jobs ==

'''Please do not run CPU- or memory-intensive tasks on the login nodes; always run them on the compute nodes. Resource-intensive processes running on login nodes may be killed at any time, without warning or notification.'''

=== Getting started ===

Jobs are submitted to the cluster using Torque and scheduled by Maui. Some documentation of the commands that are used by Torque and Maui can be found on the man pages. For example:
man qsub

To use Torque and Maui, you'll need to make sure the base [[Setting Up Your Environment#Accessing application software with modules | module]] is loaded in your shell's setup files:

module load base

Feel free to to contact us. [mailto:TRACC_Director@anl.gov Click to send email] if you need help with Torque or Maui. Note that other documentation you might find regarding Torque and Maui may not apply to the TRACC cluster because of differences in how we've configured the software.

=== Torque's view of the cluster ===

Torque and Maui are both used to schedule and run job for both the arrow and nhtsa queues.

=== Submitting jobs ===

==== Job scripts ====

The usual way of running a job with Torque is to submit a shell script. For most commercial applications we have wrapper scripts to automatically generate job scripts; see our [[TRACC Cluster Software#Supported On ARROW | application notes for our supported software]], or contact your sponsor for more info, then skip ahead to information about [[#qsub usage | qsub]].

A simple script to run an MPI job would be

#!/bin/bash

# PBS directives must precede all executable commands

cd $PBS_O_WORKDIR

## HPMPI
# mpirun -hostfile $PBS_NODEFILE ./mpi-application

## Intel MPI
# mpirun ./mpi-application

If you need to load a module in your shell script, do it like this, for scripts written in /bin/sh or /bin/bash:

. /etc/profile.d/modules.sh
module load <some-module>

or like this, if your script is written in /bin/csh or /bin/tcsh:

. /etc/profile.d/modules.csh
module load <some-module>

or like this, for Python:

import sys
sys.path.append("/soft/modules/current/init")

import envmodule
envmodule.module("load <some-module>")

or like this, for Perl:

use lib "/soft/modules/current/init";
use envmodule;
module("load <some-module");

For other languages, you can usually achieve the same effect by loading the module in your shell's configuration files (.bashrc or .cshrc).

==== Torque variables ====

* Torque sets several variables which you can use in your job scripts. See the qsub man page for the complete list.

* All Torque variables begin with "PBS", as Torque used to be known as OpenPBS (which was the precursor of PBS Pro).

* The most important Torque variables to be aware of are

PBS_O_WORKDIR

which is the working directory at the time of job submission, and

PBS_NODEFILE

which contains a list of the compute nodes which Torque has assigned your job, in the form

n001
n001
n001
n001
n001
n001
n001
n001

with each node listed once for each core.

==== qsub usage ====

The command is used to submit jobs to the cluster. qsub's behavior can be customized with command line options or script directives. Note that all script directives must precede all executable commands, so it's best to put all directives at the top of the script file.

'''For example:'''

* To use qsub to submit a 32-core job; to the batch queue which is the default queue; with a maximum walltime of 10 hours (meaning that the job will be killed after 10 hours have elapsed):

qsub -l nodes=4:ppn=8,walltime=10:00:00 jobscript

This can be specified in the jobscript with

#PBS -l nodes=4:ppn=8
#PBS -l walltime=10:00:00

in which case the job would be submitted with

qsub jobscript

* The default resource request if one isn't specified on the command line or in the jobscript is

nodes=1:ppn=1 -l walltime=5:00

meaning 1 core on 1 node, with a walltime of 5 minutes.

'''For example:'''

* For scientific applications that rely heavily on floating point operations such as STAR-CCM+, LS-DYNA, OpenFOAM, and University CFD research codes, users should specify 16 cores per node ( PPN=16 ), and not 32 since each node has a total of 16 floating point units (FPUs). Specifying 32 cores ( PPN=32 ) per compute node will not provide substantial performance gains, but will use up commerical licenses.
* To use the nhtsa queue which is not the default queue and has 64 GB nodes:
* To use qsub to submit a 64-core job with a maximum walltime of 10 hours (meaning that the job will be killed after 10 hours have elapsed):
'''Steve can you modify the following command to use the nhtsa queue.... Larry'''
qsub -l nodes=2:ppn=32,walltime=10:00:00 jobscript

This can be specified in the jobscript with

#PBS -l nodes=2:ppn=32
#PBS -l walltime=10:00:00

in which case the job would be submitted with

qsub jobscript

* The default resource request if one isn't specified on the command line or in the jobscript is

nodes=1:ppn=1 -l walltime=5:00

meaning 1 core on 1 node, with a walltime of 5 minutes.

==== Initial directory ====

Torque by default will run your script from your home directory. You can specify a different initial directory with

qsub -d ...

For example, to start in your current directory:

qsub -d `pwd`

==== Large memory jobs ====
There are two queues that are available for large memory jobs.

* batch64 queue
** 2 nodes numbered n003 and n004
** Same design as batch queue
** 64GB of RAM per node
** available for general use

* batch128 queue
** 2 nodes numbered n001 and n002
** Same design as batch queue
** 128GB of RAM per node
** available for general use

As an example, to submit your jobs to the batch128 queue:

qsub -q batch128 ...

Jobs on the nodes are limited to 72 hours.

Swap has been disabled on these nodes.

==== Short jobs ====

Weekdays from 7am CDT to 7pm CDT, two nodes are set aside for jobs on Phoenix with walltimes of 2 hours or less. Maui will automatically schedule short jobs to these nodes. No policy is currently in place for Zephyr.

=== Usage policies ===

==== Maximum walltime ====

Jobs which exceed these walltime limits will be rejected by Torque.

On Zephyr:
batch queue: 168 hours
batch64 queue: 72 hours
batch128 queue: 72 hours

On Phoenix:
quadcore queue: 168 hours
thirtytwogb queue: 72 hours

==== Quotas ====

Job which exceed the following quotas will be blocked by Maui. Blocked jobs can be seen with the command

showq -b

==== Node quotas ====

The batch64GB and batch128GB queues on Zephyr and the thirtytwogb queue on Phoenix have a limit of 1 job per user. This can also be overridden for a particular job by making the job preemptible using the Maui [[Using_the_cluster#Quality_of_Service_.28QOS.29 | QOS]] feature.

==== Queue quotas ====

The batch64GB and batch128GB queues on Zephyr and the thirtytwogb queue on Phoenix have a limit of 1 job per user. This can also be overridden for a particular job by making the job preemptible using the Maui [[Using_the_cluster#Quality_of_Service_.28QOS.29 | QOS]] feature.

=== Quality of Service (QOS) ===

In order to override the cluster's usage quotas, you can submit your jobs with a preemptible QOS. The caveat is that a preemptible job may be killed at any time, regardless of its wallclock limit, if doing so would enable another user's ordinary, non-preemptible job to run (your own non-preemptible jobs will not preempt your preemptible jobs).

To make a job preemptible, you can submit it with a QOS (quality of service) flag. For example:

qsub -W x=QOS:preemptible ... <jobscript>

You can make a submitted job (whether queued or running) preemptible with the Maui comand setqos:

setqos preemptible <jobid>

And you can make a job non-preemptible by setting its QOS to DEFAULT:

setqos DEFAULT <jobid>

You can check a job's QOS with the command [[Using_the_cluster#Job_and_cluster_info | checkjob]].

Note that because of a bug in Torque, if you want to specify a [[Using_the_cluster#Using_fewer_than_8_cores_per_node | node access policy]] and a QOS, you should not specify the QOS to Torque, but should instead submit the job without a QOS, then set the QOS using setqos.

=== How the scheduler determines which job to run next ===

Maui runs queued jobs in priority order (blocked jobs are not considered). Priority is a function of

* your recent cluster usage (the less usage, the higher the priority)

* the number of nodes requested (the more nodes, the higher the priority)

* a combination of the walltime requested (the less time, the higher the priority) and how long the job has been waiting to run

This last factor is referred to as the "expansion factor" in the Maui documentation, and is equal to

1 + time since submission / wallclock time

which, all else being equal, is designed to cause jobs to wait on average only as long as their requested walltime.

Maui also uses "backfill" to schedule short and small jobs when doing so would not interfere with higher priority jobs.

To find out how the priority of your job was calculated, use the Maui command '''diagnose''', which will show something like this:

$ diagnose -p
diagnosing job priority information (partition: ALL)

Job PRIORITY* FS( User) Serv(XFctr) Res( Proc)
Weights -------- 1( 1) 1( 1) 1( 1)

46 113 6.9( -9.0) 1.2( 1.6) 91.9(120.0)
47 77 9.5( -9.0) 1.6( 1.5) 88.9( 84.0)

Percent Contribution -------- 8.0( 8.0) 1.4( 1.4) 90.6( 90.6)

FS refers to the fairshare factor. For both jobs, the user's usage is above the target usage, making the fairshare factor negative.

The exact weight given to each factor is subject to change as we attempt to find a balance between high utilization and fairness to all users.

The diagnose command will also tell you how much you and other users have been using the cluster over the past week.

$ diagnose -f
FairShare Information

Depth: 7 intervals Interval Length: 1:00:00:00 Decay Rate: 0.75

FS Policy: UTILIZEDPS
System FS Settings: Target Usage: 0.00 Flags: 0

FSInterval % Target 0 1 2 3 4 5 6
FSWeight ------- ------- 1.0000 0.7500 0.5625 0.4219 0.3164 0.2373 0.1780
TotalUsage 100.00 ------- 23779.0 4358.0 6114.8 10451.7 10853.7 6915.1 7861.7

USER
-------------
bvannemreddy 0.00 5.00 ------- ------- ------- ------- ------- ------- -------
ac.smiyawak* 16.74 5.00 18.66 32.48 0.75 16.23 0.71 17.28 27.11
bernard 0.09 5.00 ------- ------- ------- 0.80 0.00 0.02 0.00
ac.abarsan 2.17 5.00 2.08 ------- ------- ------- ------- 17.27 8.36

The asterisk indicates that the user is above their fairshare target, and that their jobs will be given a negative fairshare weight.

=== More qsub options ===

==== Receiving mail from Torque ====

You can have Torque automatically email you when a job begins, ends, or aborts.

qsub -m <mail-options>

The <mail-options> argument consists of either the single character 'n', or one or more of 'a', 'b', 'e'.

n No mail will be sent.
a Mail will be sent when the job aborts.
b Mail will be sent when the job begins.
e Mail will be sent when the job ends.

==== Redirecting job output ====

Torque will write the standard output of your jobscript to <jobname>.o<jobnumber>, and the standard error to <jobname>.e<jobnumber>. To write them both to the standard output file, submit the job with

qsub -joe ...

or in your jobscript use

#PBS -joe

To write standard output to a different file, submit the job with

qsub -o <some-file>

or in your jobscript:

#PBS -o <some-file>

These can be combined; so, for example, submitting a job with

qsub -joe -o logfile

will cause both standard output and standard error to be written to the file 'logfile' in the directory in which the job is running.

You can see the output file using the command qpeek:

qpeek <jobid>

qpeek has several useful options:

$ qpeek -?
qpeek: Peek into a job's output spool files

Usage: qpeek [options] JOBID

Options:
-c Show all of the output file ("cat", default)
-h Show only the beginning of the output file ("head")
-t Show only the end of the output file ("tail")
-f Show only the end of the file and keep listening ("tail -f")
-# Show only # lines of output
-e Show the stderr file of the job
-o Show the stdout file of the job (default)
-? Display this help message

==== Using fewer than 8 cores per node on Phoenix and fewer than 32 cores per node on Zephyr====

'''For Phoenix'''

* To request and use fewer than 8 cores per node, specify the number of cores you need with the '''ppn''' node property:

qsub -q quadcore -l nodes=<number of nodes>:ppn=<cores per node> ...

For example, to use 4 cores on each of 8 nodes, do

qsub -q quadcore -l nodes=8:ppn=4 ...

* By default, the scheduler will assign multiple jobs to a node as long as the node has resources available, and as long as the jobs are owned by the same user. This behavior can be prevented like this:

qsub -W x=NACCESSPOLICY:SINGLEJOB ...

so the previous command would be

qsub -q quadcore -l nodes=8:ppn=4 -W x=NACCESSPOLICY:SINGLEJOB ...

(It would probably make more sense to have SINGLEJOB as the default, but Maui for some reason does not allow that policy to be overridden.)

'''For Zephyr'''

* To request and use fewer than 32 cores per node, specify the number of cores you need with the '''ppn''' node property:

qsub -q batch -l nodes=<number of nodes>:ppn=<cores per node> ...

For example, to use 4 cores on each of 8 nodes, do

qsub -q batch -l nodes=8:ppn=4 ...

* By default, the scheduler will assign multiple jobs to a node as long as the node has resources available, and as long as the jobs are owned by the same user. This behavior can be prevented like this:

qsub -W x=NACCESSPOLICY:SINGLEJOB ...

so the previous command would be

qsub -q batch -l nodes=8:ppn=4 -W x=NACCESSPOLICY:SINGLEJOB ...

(It would probably make more sense to have SINGLEJOB as the default, but Maui for some reason does not allow that policy to be overridden.)

==== Forcing jobs to run sequentially ====

* To have a job start only after another job has finished, use the '''depend''' job attribute:

qsub -W depend=afterany:<some job-id> ...

For example,

qsub -W depend=afterany:1198 ...

Many other dependency relationships are possible; please see the [http://www.clusterresources.com/torquedocs21/commands/qsub.shtml qsub man page].

==== Passing variables to a jobscript ====

By default, no environment variables are directly exported from the submission environment to the execution environment in which a job runs. This means that you'll usually need to load any modules the job requires in your job script, or in your shell configuration files, as described above, or use one of the following techniques to pass variables to a job.

* To pass all environment variables:

export name=abc; export model=neon
qsub -V ...

* To pass only specified variables:

qsub -v name=abc,model=neon ...

or

export name=abc; export model=neon
qsub -v name,model ...

* You can also specify in a job script which variables Torque should import into the execution environment from the submission environment. To import all environment variables:

#PBS -V

or to import only specified variables:

#PBS -v name,model

Note that you cannot assign values to the variables this way.

=== Job and cluster info ===

* To see a list of jobs which are running, queued, and blocked, use the Maui command showq:

showq [-r | -i | -b ]

For example,

$ showq -r
JobName S Par Effic XFactor Q User Group MHost Procs Remaining StartTime

67174 R DEF 100.00 0.0 pr joe joe n101-ib 1 00:23:32 Wed Apr 28 14:05:12
67233+ R DEF 182.60 1.1 pr jan jan n082-ib 8 14:53:30 Thu Apr 29 13:37:49
66942- R DEF 49.71 0.3 pr lin lin n056-ib 64 18:59:14 Tue Apr 27 08:41:35

The "+" indicates that the job was [[#How_the_scheduler_determines_which_job_to_run_next | backfilled]]; the "—" that the job is [[Using_the_cluster#Quality_of_Service_.28QOS.29 | preemptible]].

showq lists queued (Idle) jobs in descending [[#How_the_scheduler_determines_which_job_to_run_next | priority]] order. showq may show a job as running for up to a minute before it actually starts. To see if it's actually running in that first minute, use Torque's qstat:

qstat -a

* To see how many cores are currently free for a job of a given walltime in each queue, use qbf:

qbf -d <walltime>

* To get more detailed statistics about current cluster usage, use our command qsum:

qsum

* Use pbstop for a graphical display of cluster usage:

pbstop [-?]

* pestat will give node-by-node cluster information:

pestat [-?]

* To get a lot of info from Torque about a running or queued job:

qstat -f <job-id>

* To diagnose problems with a job, you'll need to use both the Torque command tracejob and the Maui command checkjob, and possibly the Maui command diagnose:

tracejob <job-id>
checkjob [-v] <job-id>
diagnose -j <job-id>

* If you want to find out when Maui thinks your job will run, use showstart:

showstart <job-id>

$ showstart 56
job 56 requires 32 procs for 1:40:00
Earliest start in 1:32:48 on Wed Nov 25 18:02:37
Earliest completion in 3:12:48 on Wed Nov 25 19:42:37

* If you want to find out how small and how short you need to make a job for it to run immediately in a particular queue, use the Maui command showbf:

showbf [-f <node-feature>] [-n nodecount] [-d walltime]

where <node-feature> is one of quadcore, eightgb, or thirtytwogb. ("bf" is for backfill, the algorithm used to schedule short and small jobs without interfering with higher priority jobs.) Maui unfortunately doesn't take into account all of our scheduling policies, so showbf's core counts may be too high; nor does it know about our queue walltime limits.

* We have our own utility, qbf, which summarizes the output of showbf for each queue:

$ qbf
Queue Cores Time
quadcore 112 5:49:09
quadcore 40 3:00:00:00
thirtytwogb 8 1:00:00:00

This means that a 40 core job of any duration will run immediately, but a job requesting more than 40 cores will run immediately only if it requests less than 5 hours, 49 minutes. qbf is, however, subject to the same caveat as showbf.

* To get a list of the nodes on which jobs are running (use -s to sort the list):

jobnodes [-s] <jobid> ...

$ jobnodes -s 45124 45873
n008,n015

* To get a list of the nodes on which your jobs are running (use -s to sort the list):

mynodes [-s]

$ mynodes -s
n010,n097

* To see if your application is still running on your nodes:

$ pdsh -w $(mynodes) ps

n046: PID TTY TIME CMD
n046: 13056 ? 00:00:00 bash
n046: 13717 ? 00:00:00 bash
n046: 13721 ? 00:00:00 sshd
n046: 13722 ? 00:00:00 ps

For all of these commands, only the numerical portion of the job-id needs to be specified.

==== qsub-stat ====

qsub-stat is available only on Phoenix and requires the ls-dyna/base module to be loaded.

Most of the commands for tracking your jobs and checking cluster status can be accessed via graphical application qsub-stat. Please refer to section on NoMachine to learn how to run graphical sessions on the TRACC Cluster. qsub-stat has three tabs. The first one shows percentage usage of the two queues (8GB and 32GB) and the disk space usage. Once you move mouse cursor on the meter you will get more detailed information on usage of the queues and the disk space.

[[Image:600px-Resources2.png|thumb|none|left|600px|alt=Resources tab|Resources tab]]

In the Terminal tab majority of explained previously statistics commands can be called through mouse menu. Press and hold right or left mouse button to have access to the commands' list.

[[Image:600px-Terminal2.png|thumb|none|left|600px|alt=Terminal tab|Terminal tab]]

The available commands are:

-------------------------------------------
- Cluster Usage (showq):
- showq
- showq - r (running)
- showq - b (blocked)
- Cluster Usage (qsum)
- Cluster Usage (qstat)
- Available Resources (qbf)
-------------------------------------------
- LS-DYNA Usage (qdyna)
- Madymo Usage (license server report)
- Fluent Usage (license server report)
-------------------------------------------
- Current Login Node activity (htop)
- Node activity (pbstop)
- Node activity (pestat)
-------------------------------------------
- See My Jobs
-------------------------------------------

Current usage of the cluster can also be viewed in the Job Status tab. When you press and hold right or left mouse button on a job, you will see a menu of commands that you can apply to it (Some of them will work only on your own jobs). The available commands are:

-------------------------------------------
- Show additional info:
- qstat -f
- chcekjob
- tracejob
- diagnosejob -j
- showstart
-------------------------------------------
- open standard output
- statistics on master node (htop)
- ssh to master node
-------------------------------------------
- open work directory in browser
- open work directory in terminal
-------------------------------------------
- delete job (qdel)
-------------------------------------------

[[Image:600px-Job_status2.png|thumb|none|left|600px|alt=Job Status tab|Job Status tab]]

Htop command shows live statistics on the master node of the current job.
[[Image:600px-Htop.png|thumb|none|left|600px|alt=Htop|Htop]]

=== Deleting a job ===

To delete a job, use Torque's qdel or Maui's canceljob, which simply calls qdel itself:

qdel <job-id>
canceljob <job-id>

Torque will delete the job immmediately with either command, but Maui may not know about the deletion for up to a minute, during which time it will continue to show up in the output of showq.

If either Torque or Maui tells you it can't delete the job, please send us an email and we'll delete it for you.

Only the numerical portion of the job-id needs to be specified.

=== Interactive Jobs ===

You can get interactive access to a compute node with Torque by using the -I switch to qsub:

$ qsub -I ...
qsub: waiting for job nnnn.host1 to start
qsub: job nnnn.host1 ready
$ cd $PBS_O_WORKDIR

If you want to run a GUI application on the compute node, use the -X switch:

qsub -I -X ...

For this to work, you will need to either use the [[#Graphical_Applications_using_NoMachine_NX | NX client]] to connect to the cluster, or enable trusted [[#Graphical_Applications_using_X | X11]] forwarding when you ssh to the cluster. Windows SSH clients will allow you to enable forwarding in the preferences for the connection. Under Linux and MacOS, you would do this:

$ ssh -Y login.tracc.anl.gov

Also see [[#Accessing_a_compute_node_directly | below]] for an alternative means of directly accessing computes nodes.

=== Using GUI applications on a compute node ===

If you want to use a GUI application on a compute node, you will need to enable trusted [[#Graphical_Applications_using_X | X11]] forwarding when you ssh to the cluster, and again when you ssh to the compute node (as described above). Windows SSH clients will allow you to enable forwarding in the preferences for the connection. Under Linux and MacOS, you would do this:

$ ssh -Y login.tracc.anl.gov

After logging in to a login node, you would then do this to access the compute node:

$ ssh -Y n001

If you're using the [[#Graphical_Applications_using_NoMachine_NX | NX client]] to connect to the cluster, you won't need to ssh to a login node, but you will still need to ssh to a compute node.

=== Accessing a compute node directly ===

The easiest way to do this is by having Torque give you [[#Interactive_jobs | interactive]] access to a node. But if you want to start a process running on a compute node and have it continue to run after you log out, use the method described here.

* You can access any compute node which is running one of your jobs with the command

ssh <compute-node>

You can use the command

qstat -n1 <job-id>

to find out which nodes are running <job-id>.

* If you want Torque to assign you nodes, but don't need it to run a job on your behalf, you can do (for example)

echo sleep 24h | qsub -l select=1,walltime=24:00:00

and then ssh to the node.

=== Using licensed software ===

For some of our software we have a limited number of licenses. To ensure that there will be sufficient licenses available when Maui schedules your job, you will need to submit your job under an application-specific account. Any application wrapper scripts we have will take care of this for you, but if you need to specify the account manually, you would do it like this:

* LS-DYNA:

qsub -A dyna ...

=== Debugging a job ===

Some things to keep in mind or to try when your job doesn't run as expected:

* Jobs don't inherit any environment variables from the submission environment. This means that if your job requires that PATH, LD_LIBRARY_PATH, MPI_HOME, or any other variable be set, you'll need to set it either in the job script, either directly or with a module, or in your shell configuration files.

* Try running your job interactively on a compute node, like this:

$ qsub -I ...
qsub: waiting for job nnnn.host1 to start
qsub: job nnnn.host1 ready
$ cd $PBS_O_WORKDIR

Now run your job script or start up a debugger with your program, just as you would on a login node.

* Try some simple jobs, and look at the standard output and standard error files (STDIN.o<job-id> and STDIN.e<job-id>):

echo | qsub
echo env | qsub

* If you want to see the output of your job as it's being produced, use qpeek:

qpeek <jobid>

=== Undeliverable job output ===

If Torque cannot copy the files containing the output of your job to the job's working directory, the files will eventually be copied to /mnt/gpfs/undelivered. They will be deleted after 10 days.

=== Requeuing of jobs ===

If your job dies due to a problem with the cluster, it will not automatically be requeued. If you would like it to be, do

qsub -r y ...

=== Reserving the cluster ===

Users cannot directly reserve the cluster, but if you think you might have a need for a reservation, please contact us.
[mailto:TRACC_Director@anl.gov Click to send email] as far in advance as possible.

Running Your Code

2025-04-16T22:11:17Z

Amiot: /* Reserving the cluster */

= Careful, this chapter is quite outdated and needs to be rewritten =

== Running Jobs ==

'''Please do not run CPU- or memory-intensive tasks on the login nodes; always run them on the compute nodes. Resource-intensive processes running on login nodes may be killed at any time, without warning or notification.'''

=== Getting started ===

Jobs are submitted to the cluster using Torque and scheduled by Maui. Some documentation of the commands that are used by Torque and Maui can be found on the man pages. For example:
man qsub

To use Torque and Maui, you'll need to make sure the base [[Setting Up Your Environment#Accessing application software with modules | module]] is loaded in your shell's setup files:

module load base

Feel free to to contact us. [mailto:TRACC_Director@anl.gov Click to send email] or via telephone: 630-252-8224 if you need help with Torque or Maui. Note that other documentation you might find regarding Torque and Maui may not apply to the TRACC cluster because of differences in how we've configured the software.

=== Torque's view of the cluster ===

Torque and Maui are both used to schedule and run job for both the arrow and nhtsa queues.

=== Submitting jobs ===

==== Job scripts ====

The usual way of running a job with Torque is to submit a shell script. For most commercial applications we have wrapper scripts to automatically generate job scripts; see our [[TRACC Cluster Software#Supported On ARROW | application notes for our supported software]], or contact your sponsor for more info, then skip ahead to information about [[#qsub usage | qsub]].

A simple script to run an MPI job would be

#!/bin/bash

# PBS directives must precede all executable commands

cd $PBS_O_WORKDIR

## HPMPI
# mpirun -hostfile $PBS_NODEFILE ./mpi-application

## Intel MPI
# mpirun ./mpi-application

If you need to load a module in your shell script, do it like this, for scripts written in /bin/sh or /bin/bash:

. /etc/profile.d/modules.sh
module load <some-module>

or like this, if your script is written in /bin/csh or /bin/tcsh:

. /etc/profile.d/modules.csh
module load <some-module>

or like this, for Python:

import sys
sys.path.append("/soft/modules/current/init")

import envmodule
envmodule.module("load <some-module>")

or like this, for Perl:

use lib "/soft/modules/current/init";
use envmodule;
module("load <some-module");

For other languages, you can usually achieve the same effect by loading the module in your shell's configuration files (.bashrc or .cshrc).

==== Torque variables ====

* Torque sets several variables which you can use in your job scripts. See the qsub man page for the complete list.

* All Torque variables begin with "PBS", as Torque used to be known as OpenPBS (which was the precursor of PBS Pro).

* The most important Torque variables to be aware of are

PBS_O_WORKDIR

which is the working directory at the time of job submission, and

PBS_NODEFILE

which contains a list of the compute nodes which Torque has assigned your job, in the form

n001
n001
n001
n001
n001
n001
n001
n001

with each node listed once for each core.

==== qsub usage ====

The command is used to submit jobs to the cluster. qsub's behavior can be customized with command line options or script directives. Note that all script directives must precede all executable commands, so it's best to put all directives at the top of the script file.

'''For example:'''

* To use qsub to submit a 32-core job; to the batch queue which is the default queue; with a maximum walltime of 10 hours (meaning that the job will be killed after 10 hours have elapsed):

qsub -l nodes=4:ppn=8,walltime=10:00:00 jobscript

This can be specified in the jobscript with

#PBS -l nodes=4:ppn=8
#PBS -l walltime=10:00:00

in which case the job would be submitted with

qsub jobscript

* The default resource request if one isn't specified on the command line or in the jobscript is

nodes=1:ppn=1 -l walltime=5:00

meaning 1 core on 1 node, with a walltime of 5 minutes.

'''For example:'''

* For scientific applications that rely heavily on floating point operations such as STAR-CCM+, LS-DYNA, OpenFOAM, and University CFD research codes, users should specify 16 cores per node ( PPN=16 ), and not 32 since each node has a total of 16 floating point units (FPUs). Specifying 32 cores ( PPN=32 ) per compute node will not provide substantial performance gains, but will use up commerical licenses.
* To use the nhtsa queue which is not the default queue and has 64 GB nodes:
* To use qsub to submit a 64-core job with a maximum walltime of 10 hours (meaning that the job will be killed after 10 hours have elapsed):
'''Steve can you modify the following command to use the nhtsa queue.... Larry'''
qsub -l nodes=2:ppn=32,walltime=10:00:00 jobscript

This can be specified in the jobscript with

#PBS -l nodes=2:ppn=32
#PBS -l walltime=10:00:00

in which case the job would be submitted with

qsub jobscript

* The default resource request if one isn't specified on the command line or in the jobscript is

nodes=1:ppn=1 -l walltime=5:00

meaning 1 core on 1 node, with a walltime of 5 minutes.

==== Initial directory ====

Torque by default will run your script from your home directory. You can specify a different initial directory with

qsub -d ...

For example, to start in your current directory:

qsub -d `pwd`

==== Large memory jobs ====
There are two queues that are available for large memory jobs.

* batch64 queue
** 2 nodes numbered n003 and n004
** Same design as batch queue
** 64GB of RAM per node
** available for general use

* batch128 queue
** 2 nodes numbered n001 and n002
** Same design as batch queue
** 128GB of RAM per node
** available for general use

As an example, to submit your jobs to the batch128 queue:

qsub -q batch128 ...

Jobs on the nodes are limited to 72 hours.

Swap has been disabled on these nodes.

==== Short jobs ====

Weekdays from 7am CDT to 7pm CDT, two nodes are set aside for jobs on Phoenix with walltimes of 2 hours or less. Maui will automatically schedule short jobs to these nodes. No policy is currently in place for Zephyr.

=== Usage policies ===

==== Maximum walltime ====

Jobs which exceed these walltime limits will be rejected by Torque.

On Zephyr:
batch queue: 168 hours
batch64 queue: 72 hours
batch128 queue: 72 hours

On Phoenix:
quadcore queue: 168 hours
thirtytwogb queue: 72 hours

==== Quotas ====

Job which exceed the following quotas will be blocked by Maui. Blocked jobs can be seen with the command

showq -b

==== Node quotas ====

The batch64GB and batch128GB queues on Zephyr and the thirtytwogb queue on Phoenix have a limit of 1 job per user. This can also be overridden for a particular job by making the job preemptible using the Maui [[Using_the_cluster#Quality_of_Service_.28QOS.29 | QOS]] feature.

==== Queue quotas ====

The batch64GB and batch128GB queues on Zephyr and the thirtytwogb queue on Phoenix have a limit of 1 job per user. This can also be overridden for a particular job by making the job preemptible using the Maui [[Using_the_cluster#Quality_of_Service_.28QOS.29 | QOS]] feature.

=== Quality of Service (QOS) ===

In order to override the cluster's usage quotas, you can submit your jobs with a preemptible QOS. The caveat is that a preemptible job may be killed at any time, regardless of its wallclock limit, if doing so would enable another user's ordinary, non-preemptible job to run (your own non-preemptible jobs will not preempt your preemptible jobs).

To make a job preemptible, you can submit it with a QOS (quality of service) flag. For example:

qsub -W x=QOS:preemptible ... <jobscript>

You can make a submitted job (whether queued or running) preemptible with the Maui comand setqos:

setqos preemptible <jobid>

And you can make a job non-preemptible by setting its QOS to DEFAULT:

setqos DEFAULT <jobid>

You can check a job's QOS with the command [[Using_the_cluster#Job_and_cluster_info | checkjob]].

Note that because of a bug in Torque, if you want to specify a [[Using_the_cluster#Using_fewer_than_8_cores_per_node | node access policy]] and a QOS, you should not specify the QOS to Torque, but should instead submit the job without a QOS, then set the QOS using setqos.

=== How the scheduler determines which job to run next ===

Maui runs queued jobs in priority order (blocked jobs are not considered). Priority is a function of

* your recent cluster usage (the less usage, the higher the priority)

* the number of nodes requested (the more nodes, the higher the priority)

* a combination of the walltime requested (the less time, the higher the priority) and how long the job has been waiting to run

This last factor is referred to as the "expansion factor" in the Maui documentation, and is equal to

1 + time since submission / wallclock time

which, all else being equal, is designed to cause jobs to wait on average only as long as their requested walltime.

Maui also uses "backfill" to schedule short and small jobs when doing so would not interfere with higher priority jobs.

To find out how the priority of your job was calculated, use the Maui command '''diagnose''', which will show something like this:

$ diagnose -p
diagnosing job priority information (partition: ALL)

Job PRIORITY* FS( User) Serv(XFctr) Res( Proc)
Weights -------- 1( 1) 1( 1) 1( 1)

46 113 6.9( -9.0) 1.2( 1.6) 91.9(120.0)
47 77 9.5( -9.0) 1.6( 1.5) 88.9( 84.0)

Percent Contribution -------- 8.0( 8.0) 1.4( 1.4) 90.6( 90.6)

FS refers to the fairshare factor. For both jobs, the user's usage is above the target usage, making the fairshare factor negative.

The exact weight given to each factor is subject to change as we attempt to find a balance between high utilization and fairness to all users.

The diagnose command will also tell you how much you and other users have been using the cluster over the past week.

$ diagnose -f
FairShare Information

Depth: 7 intervals Interval Length: 1:00:00:00 Decay Rate: 0.75

FS Policy: UTILIZEDPS
System FS Settings: Target Usage: 0.00 Flags: 0

FSInterval % Target 0 1 2 3 4 5 6
FSWeight ------- ------- 1.0000 0.7500 0.5625 0.4219 0.3164 0.2373 0.1780
TotalUsage 100.00 ------- 23779.0 4358.0 6114.8 10451.7 10853.7 6915.1 7861.7

USER
-------------
bvannemreddy 0.00 5.00 ------- ------- ------- ------- ------- ------- -------
ac.smiyawak* 16.74 5.00 18.66 32.48 0.75 16.23 0.71 17.28 27.11
bernard 0.09 5.00 ------- ------- ------- 0.80 0.00 0.02 0.00
ac.abarsan 2.17 5.00 2.08 ------- ------- ------- ------- 17.27 8.36

The asterisk indicates that the user is above their fairshare target, and that their jobs will be given a negative fairshare weight.

=== More qsub options ===

==== Receiving mail from Torque ====

You can have Torque automatically email you when a job begins, ends, or aborts.

qsub -m <mail-options>

The <mail-options> argument consists of either the single character 'n', or one or more of 'a', 'b', 'e'.

n No mail will be sent.
a Mail will be sent when the job aborts.
b Mail will be sent when the job begins.
e Mail will be sent when the job ends.

==== Redirecting job output ====

Torque will write the standard output of your jobscript to <jobname>.o<jobnumber>, and the standard error to <jobname>.e<jobnumber>. To write them both to the standard output file, submit the job with

qsub -joe ...

or in your jobscript use

#PBS -joe

To write standard output to a different file, submit the job with

qsub -o <some-file>

or in your jobscript:

#PBS -o <some-file>

These can be combined; so, for example, submitting a job with

qsub -joe -o logfile

will cause both standard output and standard error to be written to the file 'logfile' in the directory in which the job is running.

You can see the output file using the command qpeek:

qpeek <jobid>

qpeek has several useful options:

$ qpeek -?
qpeek: Peek into a job's output spool files

Usage: qpeek [options] JOBID

Options:
-c Show all of the output file ("cat", default)
-h Show only the beginning of the output file ("head")
-t Show only the end of the output file ("tail")
-f Show only the end of the file and keep listening ("tail -f")
-# Show only # lines of output
-e Show the stderr file of the job
-o Show the stdout file of the job (default)
-? Display this help message

==== Using fewer than 8 cores per node on Phoenix and fewer than 32 cores per node on Zephyr====

'''For Phoenix'''

* To request and use fewer than 8 cores per node, specify the number of cores you need with the '''ppn''' node property:

qsub -q quadcore -l nodes=<number of nodes>:ppn=<cores per node> ...

For example, to use 4 cores on each of 8 nodes, do

qsub -q quadcore -l nodes=8:ppn=4 ...

* By default, the scheduler will assign multiple jobs to a node as long as the node has resources available, and as long as the jobs are owned by the same user. This behavior can be prevented like this:

qsub -W x=NACCESSPOLICY:SINGLEJOB ...

so the previous command would be

qsub -q quadcore -l nodes=8:ppn=4 -W x=NACCESSPOLICY:SINGLEJOB ...

(It would probably make more sense to have SINGLEJOB as the default, but Maui for some reason does not allow that policy to be overridden.)

'''For Zephyr'''

* To request and use fewer than 32 cores per node, specify the number of cores you need with the '''ppn''' node property:

qsub -q batch -l nodes=<number of nodes>:ppn=<cores per node> ...

For example, to use 4 cores on each of 8 nodes, do

qsub -q batch -l nodes=8:ppn=4 ...

* By default, the scheduler will assign multiple jobs to a node as long as the node has resources available, and as long as the jobs are owned by the same user. This behavior can be prevented like this:

qsub -W x=NACCESSPOLICY:SINGLEJOB ...

so the previous command would be

qsub -q batch -l nodes=8:ppn=4 -W x=NACCESSPOLICY:SINGLEJOB ...

(It would probably make more sense to have SINGLEJOB as the default, but Maui for some reason does not allow that policy to be overridden.)

==== Forcing jobs to run sequentially ====

* To have a job start only after another job has finished, use the '''depend''' job attribute:

qsub -W depend=afterany:<some job-id> ...

For example,

qsub -W depend=afterany:1198 ...

Many other dependency relationships are possible; please see the [http://www.clusterresources.com/torquedocs21/commands/qsub.shtml qsub man page].

==== Passing variables to a jobscript ====

By default, no environment variables are directly exported from the submission environment to the execution environment in which a job runs. This means that you'll usually need to load any modules the job requires in your job script, or in your shell configuration files, as described above, or use one of the following techniques to pass variables to a job.

* To pass all environment variables:

export name=abc; export model=neon
qsub -V ...

* To pass only specified variables:

qsub -v name=abc,model=neon ...

or

export name=abc; export model=neon
qsub -v name,model ...

* You can also specify in a job script which variables Torque should import into the execution environment from the submission environment. To import all environment variables:

#PBS -V

or to import only specified variables:

#PBS -v name,model

Note that you cannot assign values to the variables this way.

=== Job and cluster info ===

* To see a list of jobs which are running, queued, and blocked, use the Maui command showq:

showq [-r | -i | -b ]

For example,

$ showq -r
JobName S Par Effic XFactor Q User Group MHost Procs Remaining StartTime

67174 R DEF 100.00 0.0 pr joe joe n101-ib 1 00:23:32 Wed Apr 28 14:05:12
67233+ R DEF 182.60 1.1 pr jan jan n082-ib 8 14:53:30 Thu Apr 29 13:37:49
66942- R DEF 49.71 0.3 pr lin lin n056-ib 64 18:59:14 Tue Apr 27 08:41:35

The "+" indicates that the job was [[#How_the_scheduler_determines_which_job_to_run_next | backfilled]]; the "—" that the job is [[Using_the_cluster#Quality_of_Service_.28QOS.29 | preemptible]].

showq lists queued (Idle) jobs in descending [[#How_the_scheduler_determines_which_job_to_run_next | priority]] order. showq may show a job as running for up to a minute before it actually starts. To see if it's actually running in that first minute, use Torque's qstat:

qstat -a

* To see how many cores are currently free for a job of a given walltime in each queue, use qbf:

qbf -d <walltime>

* To get more detailed statistics about current cluster usage, use our command qsum:

qsum

* Use pbstop for a graphical display of cluster usage:

pbstop [-?]

* pestat will give node-by-node cluster information:

pestat [-?]

* To get a lot of info from Torque about a running or queued job:

qstat -f <job-id>

* To diagnose problems with a job, you'll need to use both the Torque command tracejob and the Maui command checkjob, and possibly the Maui command diagnose:

tracejob <job-id>
checkjob [-v] <job-id>
diagnose -j <job-id>

* If you want to find out when Maui thinks your job will run, use showstart:

showstart <job-id>

$ showstart 56
job 56 requires 32 procs for 1:40:00
Earliest start in 1:32:48 on Wed Nov 25 18:02:37
Earliest completion in 3:12:48 on Wed Nov 25 19:42:37

* If you want to find out how small and how short you need to make a job for it to run immediately in a particular queue, use the Maui command showbf:

showbf [-f <node-feature>] [-n nodecount] [-d walltime]

where <node-feature> is one of quadcore, eightgb, or thirtytwogb. ("bf" is for backfill, the algorithm used to schedule short and small jobs without interfering with higher priority jobs.) Maui unfortunately doesn't take into account all of our scheduling policies, so showbf's core counts may be too high; nor does it know about our queue walltime limits.

* We have our own utility, qbf, which summarizes the output of showbf for each queue:

$ qbf
Queue Cores Time
quadcore 112 5:49:09
quadcore 40 3:00:00:00
thirtytwogb 8 1:00:00:00

This means that a 40 core job of any duration will run immediately, but a job requesting more than 40 cores will run immediately only if it requests less than 5 hours, 49 minutes. qbf is, however, subject to the same caveat as showbf.

* To get a list of the nodes on which jobs are running (use -s to sort the list):

jobnodes [-s] <jobid> ...

$ jobnodes -s 45124 45873
n008,n015

* To get a list of the nodes on which your jobs are running (use -s to sort the list):

mynodes [-s]

$ mynodes -s
n010,n097

* To see if your application is still running on your nodes:

$ pdsh -w $(mynodes) ps

n046: PID TTY TIME CMD
n046: 13056 ? 00:00:00 bash
n046: 13717 ? 00:00:00 bash
n046: 13721 ? 00:00:00 sshd
n046: 13722 ? 00:00:00 ps

For all of these commands, only the numerical portion of the job-id needs to be specified.

==== qsub-stat ====

qsub-stat is available only on Phoenix and requires the ls-dyna/base module to be loaded.

Most of the commands for tracking your jobs and checking cluster status can be accessed via graphical application qsub-stat. Please refer to section on NoMachine to learn how to run graphical sessions on the TRACC Cluster. qsub-stat has three tabs. The first one shows percentage usage of the two queues (8GB and 32GB) and the disk space usage. Once you move mouse cursor on the meter you will get more detailed information on usage of the queues and the disk space.

[[Image:600px-Resources2.png|thumb|none|left|600px|alt=Resources tab|Resources tab]]

In the Terminal tab majority of explained previously statistics commands can be called through mouse menu. Press and hold right or left mouse button to have access to the commands' list.

[[Image:600px-Terminal2.png|thumb|none|left|600px|alt=Terminal tab|Terminal tab]]

The available commands are:

-------------------------------------------
- Cluster Usage (showq):
- showq
- showq - r (running)
- showq - b (blocked)
- Cluster Usage (qsum)
- Cluster Usage (qstat)
- Available Resources (qbf)
-------------------------------------------
- LS-DYNA Usage (qdyna)
- Madymo Usage (license server report)
- Fluent Usage (license server report)
-------------------------------------------
- Current Login Node activity (htop)
- Node activity (pbstop)
- Node activity (pestat)
-------------------------------------------
- See My Jobs
-------------------------------------------

Current usage of the cluster can also be viewed in the Job Status tab. When you press and hold right or left mouse button on a job, you will see a menu of commands that you can apply to it (Some of them will work only on your own jobs). The available commands are:

-------------------------------------------
- Show additional info:
- qstat -f
- chcekjob
- tracejob
- diagnosejob -j
- showstart
-------------------------------------------
- open standard output
- statistics on master node (htop)
- ssh to master node
-------------------------------------------
- open work directory in browser
- open work directory in terminal
-------------------------------------------
- delete job (qdel)
-------------------------------------------

[[Image:600px-Job_status2.png|thumb|none|left|600px|alt=Job Status tab|Job Status tab]]

Htop command shows live statistics on the master node of the current job.
[[Image:600px-Htop.png|thumb|none|left|600px|alt=Htop|Htop]]

=== Deleting a job ===

To delete a job, use Torque's qdel or Maui's canceljob, which simply calls qdel itself:

qdel <job-id>
canceljob <job-id>

Torque will delete the job immmediately with either command, but Maui may not know about the deletion for up to a minute, during which time it will continue to show up in the output of showq.

If either Torque or Maui tells you it can't delete the job, please send us an email and we'll delete it for you.

Only the numerical portion of the job-id needs to be specified.

=== Interactive Jobs ===

You can get interactive access to a compute node with Torque by using the -I switch to qsub:

$ qsub -I ...
qsub: waiting for job nnnn.host1 to start
qsub: job nnnn.host1 ready
$ cd $PBS_O_WORKDIR

If you want to run a GUI application on the compute node, use the -X switch:

qsub -I -X ...

For this to work, you will need to either use the [[#Graphical_Applications_using_NoMachine_NX | NX client]] to connect to the cluster, or enable trusted [[#Graphical_Applications_using_X | X11]] forwarding when you ssh to the cluster. Windows SSH clients will allow you to enable forwarding in the preferences for the connection. Under Linux and MacOS, you would do this:

$ ssh -Y login.tracc.anl.gov

Also see [[#Accessing_a_compute_node_directly | below]] for an alternative means of directly accessing computes nodes.

=== Using GUI applications on a compute node ===

If you want to use a GUI application on a compute node, you will need to enable trusted [[#Graphical_Applications_using_X | X11]] forwarding when you ssh to the cluster, and again when you ssh to the compute node (as described above). Windows SSH clients will allow you to enable forwarding in the preferences for the connection. Under Linux and MacOS, you would do this:

$ ssh -Y login.tracc.anl.gov

After logging in to a login node, you would then do this to access the compute node:

$ ssh -Y n001

If you're using the [[#Graphical_Applications_using_NoMachine_NX | NX client]] to connect to the cluster, you won't need to ssh to a login node, but you will still need to ssh to a compute node.

=== Accessing a compute node directly ===

The easiest way to do this is by having Torque give you [[#Interactive_jobs | interactive]] access to a node. But if you want to start a process running on a compute node and have it continue to run after you log out, use the method described here.

* You can access any compute node which is running one of your jobs with the command

ssh <compute-node>

You can use the command

qstat -n1 <job-id>

to find out which nodes are running <job-id>.

* If you want Torque to assign you nodes, but don't need it to run a job on your behalf, you can do (for example)

echo sleep 24h | qsub -l select=1,walltime=24:00:00

and then ssh to the node.

=== Using licensed software ===

For some of our software we have a limited number of licenses. To ensure that there will be sufficient licenses available when Maui schedules your job, you will need to submit your job under an application-specific account. Any application wrapper scripts we have will take care of this for you, but if you need to specify the account manually, you would do it like this:

* LS-DYNA:

qsub -A dyna ...

=== Debugging a job ===

Some things to keep in mind or to try when your job doesn't run as expected:

* Jobs don't inherit any environment variables from the submission environment. This means that if your job requires that PATH, LD_LIBRARY_PATH, MPI_HOME, or any other variable be set, you'll need to set it either in the job script, either directly or with a module, or in your shell configuration files.

* Try running your job interactively on a compute node, like this:

$ qsub -I ...
qsub: waiting for job nnnn.host1 to start
qsub: job nnnn.host1 ready
$ cd $PBS_O_WORKDIR

Now run your job script or start up a debugger with your program, just as you would on a login node.

* Try some simple jobs, and look at the standard output and standard error files (STDIN.o<job-id> and STDIN.e<job-id>):

echo | qsub
echo env | qsub

* If you want to see the output of your job as it's being produced, use qpeek:

qpeek <jobid>

=== Undeliverable job output ===

If Torque cannot copy the files containing the output of your job to the job's working directory, the files will eventually be copied to /mnt/gpfs/undelivered. They will be deleted after 10 days.

=== Requeuing of jobs ===

If your job dies due to a problem with the cluster, it will not automatically be requeued. If you would like it to be, do

qsub -r y ...

=== Reserving the cluster ===

Users cannot directly reserve the cluster, but if you think you might have a need for a reservation, please contact us.
[mailto:TRACC_Director@anl.gov Click to send email] as far in advance as possible.

Graphical Applications

2025-04-16T22:09:20Z

Amiot: /* Graphical Applications Using NoMachine NX */

==General Graphic Application Comments==

This chapter describes the use of graphical user interfaces to bring the Linux desktop to your local machine. This gives you the ability to work with graphical applications, browse and edit files in a graphical manner that is similar to your usual work patterns, and establish remote sessions that stay alive while you disconnect from the cluster. You can of course fully log out when appropriate, but by default, the session stays alive until you reconnect at a later time. This allows you to keep interactive applications running for very long times if needed.

There are three ways to connect to Arrow to run graphic applications. These are:

* Running the NoMachine NX client
* Running the X2Go client
* Running X and using an SSH terminal client to forward the X protocol

Running NoMachine and X2Go are the most efficient because they transfer information between the cluster and the client using data compression and a special protocol to minimize data transmissions.

Note: When using the third option, you will not have a full Linux desktop on your screen. Instead, Linux applications can bring up your user interface on your client for interaction. This is much slower than the other two options, but may be useful for your specific application. In this case, all rendering is done on your client rather than on the remote server.

Let's assume that you have an X server running on your machine, and that you have the ssh command available. In that case, you would do the following:

ssh -X userid@login1.arrow.tracc.anl.gov
enter your password
xeyes

This last command is just for testing. You should see a set of eyes that can be moved around using your mouse. If this works, then your graphical application scan be used remotely.

==Graphical Applications Using NoMachine NX==

To provide for the remote use of graphical applications on the cluster, we've installed [http://www.nomachine.com NoMachine NX] on the login hosts. This method allow you to establish a remote desktop on the login machine, which stays alive even when you disconnect from it. You can of course log out of that connection, but by default, the desktop stays alive, allowing you to keep interactive applications running while disconnected. To use this software,

* download and install the free [http://www.nomachine.com/download.php NX Client].
* connect to the TRACC cluster using one of the login nodes:
** login1.arrow.tracc.anl.gov
** login2.arrow.tracc.anl.gov
** login3.arrow.tracc.anl.gov
** login4.arrow.tracc.anl.gov
* The NX protocol for connections is not supported. Please choose SSH from the pull-down menu.
* When connecting the very first time, you are prompted to confirm that you are connecting to the correct server, showing a server key. Confirm and continue.
* When doing this for the first time, create a new Linux Desktop. This will be shown in the future when connecting again.
* Please remember that you need a proper account, password, and that we need to have your public IP address on file for all of this to work properly.



If you have problems, please contact us by [mailto:TRACC_Director@anl.gov clicking to send email].

=== Terminating and Disconnecting ===

When you quit an NX session, you can either terminate it or disconnect from it. Terminating it will close all your running programs. If you disconnect, you can reconnect later to the same session. Interactive applications will continue running while disconnected, so this is a convenient way to occasionally check on progress, or to work with long running applications despite possible network issues.

If you close the NX client window, you will be simply disconnected and can resume this connection at a later time. To fully log out, use the Linux pull-down command at the top left of your screen, and choose the bottom entry to log out.

=== Reconnecting ===

If you want to reconnect to an NX session, you'll have to log in to the same login node on which it was started.

=== Problems ===

We will post trouble-shooting instructions in this section if ever needed. The latest versions are very stable.

==Graphical Applications Using X2Go==

X2Go can be installed by following the [https://wiki.x2go.org/doku.php/doc:installation:x2goclient instructions here.]. Instructions are provided for Windows, OSX, and a number of other client OSs.

After installing the client software, create a new connection with the following settings

* connection to one of our login servers
** login1.arrow.tracc.anl.gov
** login2.arrow.tracc.anl.gov
** login3.arrow.tracc.anl.gov
** login4.arrow.tracc.anl.gov
* account name and password of course
* IMPORTANT: from the pull-down menu, choose the XFCE desktop
* Also, we recommend turning off remote audio (it won't work anyway)

* on Macs: It should be noticed that if you don't already have XQuartz installed on OSX, you will need to do that before installing X2Go.
* A general X2Go tutorial can be [https://wiki.anl.gov/wiki_traccwiki/index.php?title=Graphical_Applications&action=editfound here.]

==Graphical Applications Using X==
[http://sourceforge.net/projects/xming xming] is a convenient and easy way to install an X server for Windows. However, if you're doing pre- or post-processing that requires sending a lot of graphics over the network, we recommend NoMachine NX or X2Go because their compression of the graphics yields a significant speed advantage over X. Also, rendering is generally perform on the server side rather than your local machine, speeding things up. In addition, the X protocol is very chatty, and performs poorly over the SSH encryption channel.

*To install X on a Mac, refer to this [https://princetonuniversity.github.io/PUbootcamp/ssh-instructions site ]
* The Mac uses XQuart to implement X applications and should be installed if not alread installed on your version of OSX. To bring up the window, connect to the cluster using the following (-X or -Y is required):
ssh -Y userid@login1.arrow.tracc.anl.gov

TRACC Policies

2025-04-14T21:53:59Z

Amiot:

==Application Restrictions==
Parallel jobs must be run on the compute nodes using Torque. Multicore jobs on the login nodes may be killed without notice.

==Data Retention==
* Thirty days after the expiration of your project, your files will be deleted from the TRACC cluster and from the TRACC backup system.

==Email==
* If we cannot contact you at the email address which Argonne lists as your contact address, your account will be locked until you provide us with a valid address.

==Login Nodes==
* Resource-intensive jobs on the login nodes may be killed at any time, without warning or notification.
* Repeated violation of this policy will result in revocation of your account.

TRACC Cluster Software

2025-04-14T21:53:23Z

Amiot:

The following lists the software that is supported on ARROW and the software we are considering for support in the future. Click on the software for Application Notes.
==Supported On ARROW==

[[LS-DYNA | * LS-DYNA]]

[[LS-PrePost | * LS-PrePost]]

[[LS-OPT | * LS-OPT]]

[[OpenFOAM | * OpenFOAM]]

[[Python | * Python]]

[[STAR-CCM+ | * STAR-CCM+]]

==Under Investigation==

<nowiki>* MATLAB</nowiki>

<nowiki>* COMSOL</nowiki>

<nowiki>* Abaqus</nowiki>

Graphical Applications

2025-04-14T21:52:30Z

Amiot:

==General Graphic Application Comments==

This chapter describes the use of graphical user interfaces to bring the Linux desktop to your local machine. This gives you the ability to work with graphical applications, browse and edit files in a graphical manner that is similar to your usual work patterns, and establish remote sessions that stay alive while you disconnect from the cluster. You can of course fully log out when appropriate, but by default, the session stays alive until you reconnect at a later time. This allows you to keep interactive applications running for very long times if needed.

There are three ways to connect to Arrow to run graphic applications. These are:

* Running the NoMachine NX client
* Running the X2Go client
* Running X and using an SSH terminal client to forward the X protocol

Running NoMachine and X2Go are the most efficient because they transfer information between the cluster and the client using data compression and a special protocol to minimize data transmissions.

Note: When using the third option, you will not have a full Linux desktop on your screen. Instead, Linux applications can bring up your user interface on your client for interaction. This is much slower than the other two options, but may be useful for your specific application. In this case, all rendering is done on your client rather than on the remote server.

Let's assume that you have an X server running on your machine, and that you have the ssh command available. In that case, you would do the following:

ssh -X userid@login1.arrow.tracc.anl.gov
enter your password
xeyes

This last command is just for testing. You should see a set of eyes that can be moved around using your mouse. If this works, then your graphical application scan be used remotely.

==Graphical Applications Using NoMachine NX==

To provide for the remote use of graphical applications on the cluster, we've installed [http://www.nomachine.com NoMachine NX] on the login hosts. This method allow you to establish a remote desktop on the login machine, which stays alive even when you disconnect from it. You can of course log out of that connection, but by default, the desktop stays alive, allowing you to keep interactive applications running while disconnected. To use this software,

* download and install the free [http://www.nomachine.com/download.php NX Client].
* connect to the TRACC cluster using one of the login nodes:
** login1.arrow.tracc.anl.gov
** login2.arrow.tracc.anl.gov
** login3.arrow.tracc.anl.gov
** login4.arrow.tracc.anl.gov
* The NX protocol for connections is not supported. Please choose SSH from the pull-down menu.
* When connecting the very first time, you are prompted to confirm that you are connecting to the correct server, showing a server key. Confirm and continue.
* When doing this for the first time, create a new Linux Desktop. This will be shown in the future when connecting again.
* Please remember that you need a proper account, password, and that we need to have your public IP address on file for all of this to work properly.



If you have problems, please contact us by [mailto:TRACC_Director@anl.gov clicking to send email] or via telephone at 630-252-8224.

=== Terminating and Disconnecting ===

When you quit an NX session, you can either terminate it or disconnect from it. Terminating it will close all your running programs. If you disconnect, you can reconnect later to the same session. Interactive applications will continue running while disconnected, so this is a convenient way to occasionally check on progress, or to work with long running applications despite possible network issues.

If you close the NX client window, you will be simply disconnected and can resume this connection at a later time. To fully log out, use the Linux pull-down command at the top left of your screen, and choose the bottom entry to log out.

=== Reconnecting ===

If you want to reconnect to an NX session, you'll have to log in to the same login node on which it was started.

=== Problems ===

We will post trouble-shooting instructions in this section if ever needed. The latest versions are very stable.

==Graphical Applications Using X2Go==

X2Go can be installed by following the [https://wiki.x2go.org/doku.php/doc:installation:x2goclient instructions here.]. Instructions are provided for Windows, OSX, and a number of other client OSs.

After installing the client software, create a new connection with the following settings

* connection to one of our login servers
** login1.arrow.tracc.anl.gov
** login2.arrow.tracc.anl.gov
** login3.arrow.tracc.anl.gov
** login4.arrow.tracc.anl.gov
* account name and password of course
* IMPORTANT: from the pull-down menu, choose the XFCE desktop
* Also, we recommend turning off remote audio (it won't work anyway)

* on Macs: It should be noticed that if you don't already have XQuartz installed on OSX, you will need to do that before installing X2Go.
* A general X2Go tutorial can be [https://wiki.anl.gov/wiki_traccwiki/index.php?title=Graphical_Applications&action=editfound here.]

==Graphical Applications Using X==
[http://sourceforge.net/projects/xming xming] is a convenient and easy way to install an X server for Windows. However, if you're doing pre- or post-processing that requires sending a lot of graphics over the network, we recommend NoMachine NX or X2Go because their compression of the graphics yields a significant speed advantage over X. Also, rendering is generally perform on the server side rather than your local machine, speeding things up. In addition, the X protocol is very chatty, and performs poorly over the SSH encryption channel.

*To install X on a Mac, refer to this [https://princetonuniversity.github.io/PUbootcamp/ssh-instructions site ]
* The Mac uses XQuart to implement X applications and should be installed if not alread installed on your version of OSX. To bring up the window, connect to the cluster using the following (-X or -Y is required):
ssh -Y userid@login1.arrow.tracc.anl.gov

Running Your Code

2025-04-14T21:51:54Z

Amiot:

= Careful, this chapter is quite outdated and needs to be rewritten =

== Running Jobs ==

'''Please do not run CPU- or memory-intensive tasks on the login nodes; always run them on the compute nodes. Resource-intensive processes running on login nodes may be killed at any time, without warning or notification.'''

=== Getting started ===

Jobs are submitted to the cluster using Torque and scheduled by Maui. Some documentation of the commands that are used by Torque and Maui can be found on the man pages. For example:
man qsub

To use Torque and Maui, you'll need to make sure the base [[Setting Up Your Environment#Accessing application software with modules | module]] is loaded in your shell's setup files:

module load base

Feel free to to contact us. [mailto:TRACC_Director@anl.gov Click to send email] or via telephone: 630-252-8224 if you need help with Torque or Maui. Note that other documentation you might find regarding Torque and Maui may not apply to the TRACC cluster because of differences in how we've configured the software.

=== Torque's view of the cluster ===

Torque and Maui are both used to schedule and run job for both the arrow and nhtsa queues.

=== Submitting jobs ===

==== Job scripts ====

The usual way of running a job with Torque is to submit a shell script. For most commercial applications we have wrapper scripts to automatically generate job scripts; see our [[TRACC Cluster Software#Supported On ARROW | application notes for our supported software]], or contact your sponsor for more info, then skip ahead to information about [[#qsub usage | qsub]].

A simple script to run an MPI job would be

#!/bin/bash

# PBS directives must precede all executable commands

cd $PBS_O_WORKDIR

## HPMPI
# mpirun -hostfile $PBS_NODEFILE ./mpi-application

## Intel MPI
# mpirun ./mpi-application

If you need to load a module in your shell script, do it like this, for scripts written in /bin/sh or /bin/bash:

. /etc/profile.d/modules.sh
module load <some-module>

or like this, if your script is written in /bin/csh or /bin/tcsh:

. /etc/profile.d/modules.csh
module load <some-module>

or like this, for Python:

import sys
sys.path.append("/soft/modules/current/init")

import envmodule
envmodule.module("load <some-module>")

or like this, for Perl:

use lib "/soft/modules/current/init";
use envmodule;
module("load <some-module");

For other languages, you can usually achieve the same effect by loading the module in your shell's configuration files (.bashrc or .cshrc).

==== Torque variables ====

* Torque sets several variables which you can use in your job scripts. See the qsub man page for the complete list.

* All Torque variables begin with "PBS", as Torque used to be known as OpenPBS (which was the precursor of PBS Pro).

* The most important Torque variables to be aware of are

PBS_O_WORKDIR

which is the working directory at the time of job submission, and

PBS_NODEFILE

which contains a list of the compute nodes which Torque has assigned your job, in the form

n001
n001
n001
n001
n001
n001
n001
n001

with each node listed once for each core.

==== qsub usage ====

The command is used to submit jobs to the cluster. qsub's behavior can be customized with command line options or script directives. Note that all script directives must precede all executable commands, so it's best to put all directives at the top of the script file.

'''For example:'''

* To use qsub to submit a 32-core job; to the batch queue which is the default queue; with a maximum walltime of 10 hours (meaning that the job will be killed after 10 hours have elapsed):

qsub -l nodes=4:ppn=8,walltime=10:00:00 jobscript

This can be specified in the jobscript with

#PBS -l nodes=4:ppn=8
#PBS -l walltime=10:00:00

in which case the job would be submitted with

qsub jobscript

* The default resource request if one isn't specified on the command line or in the jobscript is

nodes=1:ppn=1 -l walltime=5:00

meaning 1 core on 1 node, with a walltime of 5 minutes.

'''For example:'''

* For scientific applications that rely heavily on floating point operations such as STAR-CCM+, LS-DYNA, OpenFOAM, and University CFD research codes, users should specify 16 cores per node ( PPN=16 ), and not 32 since each node has a total of 16 floating point units (FPUs). Specifying 32 cores ( PPN=32 ) per compute node will not provide substantial performance gains, but will use up commerical licenses.
* To use the nhtsa queue which is not the default queue and has 64 GB nodes:
* To use qsub to submit a 64-core job with a maximum walltime of 10 hours (meaning that the job will be killed after 10 hours have elapsed):
'''Steve can you modify the following command to use the nhtsa queue.... Larry'''
qsub -l nodes=2:ppn=32,walltime=10:00:00 jobscript

This can be specified in the jobscript with

#PBS -l nodes=2:ppn=32
#PBS -l walltime=10:00:00

in which case the job would be submitted with

qsub jobscript

* The default resource request if one isn't specified on the command line or in the jobscript is

nodes=1:ppn=1 -l walltime=5:00

meaning 1 core on 1 node, with a walltime of 5 minutes.

==== Initial directory ====

Torque by default will run your script from your home directory. You can specify a different initial directory with

qsub -d ...

For example, to start in your current directory:

qsub -d `pwd`

==== Large memory jobs ====
There are two queues that are available for large memory jobs.

* batch64 queue
** 2 nodes numbered n003 and n004
** Same design as batch queue
** 64GB of RAM per node
** available for general use

* batch128 queue
** 2 nodes numbered n001 and n002
** Same design as batch queue
** 128GB of RAM per node
** available for general use

As an example, to submit your jobs to the batch128 queue:

qsub -q batch128 ...

Jobs on the nodes are limited to 72 hours.

Swap has been disabled on these nodes.

==== Short jobs ====

Weekdays from 7am CDT to 7pm CDT, two nodes are set aside for jobs on Phoenix with walltimes of 2 hours or less. Maui will automatically schedule short jobs to these nodes. No policy is currently in place for Zephyr.

=== Usage policies ===

==== Maximum walltime ====

Jobs which exceed these walltime limits will be rejected by Torque.

On Zephyr:
batch queue: 168 hours
batch64 queue: 72 hours
batch128 queue: 72 hours

On Phoenix:
quadcore queue: 168 hours
thirtytwogb queue: 72 hours

==== Quotas ====

Job which exceed the following quotas will be blocked by Maui. Blocked jobs can be seen with the command

showq -b

==== Node quotas ====

The batch64GB and batch128GB queues on Zephyr and the thirtytwogb queue on Phoenix have a limit of 1 job per user. This can also be overridden for a particular job by making the job preemptible using the Maui [[Using_the_cluster#Quality_of_Service_.28QOS.29 | QOS]] feature.

==== Queue quotas ====

The batch64GB and batch128GB queues on Zephyr and the thirtytwogb queue on Phoenix have a limit of 1 job per user. This can also be overridden for a particular job by making the job preemptible using the Maui [[Using_the_cluster#Quality_of_Service_.28QOS.29 | QOS]] feature.

=== Quality of Service (QOS) ===

In order to override the cluster's usage quotas, you can submit your jobs with a preemptible QOS. The caveat is that a preemptible job may be killed at any time, regardless of its wallclock limit, if doing so would enable another user's ordinary, non-preemptible job to run (your own non-preemptible jobs will not preempt your preemptible jobs).

To make a job preemptible, you can submit it with a QOS (quality of service) flag. For example:

qsub -W x=QOS:preemptible ... <jobscript>

You can make a submitted job (whether queued or running) preemptible with the Maui comand setqos:

setqos preemptible <jobid>

And you can make a job non-preemptible by setting its QOS to DEFAULT:

setqos DEFAULT <jobid>

You can check a job's QOS with the command [[Using_the_cluster#Job_and_cluster_info | checkjob]].

Note that because of a bug in Torque, if you want to specify a [[Using_the_cluster#Using_fewer_than_8_cores_per_node | node access policy]] and a QOS, you should not specify the QOS to Torque, but should instead submit the job without a QOS, then set the QOS using setqos.

=== How the scheduler determines which job to run next ===

Maui runs queued jobs in priority order (blocked jobs are not considered). Priority is a function of

* your recent cluster usage (the less usage, the higher the priority)

* the number of nodes requested (the more nodes, the higher the priority)

* a combination of the walltime requested (the less time, the higher the priority) and how long the job has been waiting to run

This last factor is referred to as the "expansion factor" in the Maui documentation, and is equal to

1 + time since submission / wallclock time

which, all else being equal, is designed to cause jobs to wait on average only as long as their requested walltime.

Maui also uses "backfill" to schedule short and small jobs when doing so would not interfere with higher priority jobs.

To find out how the priority of your job was calculated, use the Maui command '''diagnose''', which will show something like this:

$ diagnose -p
diagnosing job priority information (partition: ALL)

Job PRIORITY* FS( User) Serv(XFctr) Res( Proc)
Weights -------- 1( 1) 1( 1) 1( 1)

46 113 6.9( -9.0) 1.2( 1.6) 91.9(120.0)
47 77 9.5( -9.0) 1.6( 1.5) 88.9( 84.0)

Percent Contribution -------- 8.0( 8.0) 1.4( 1.4) 90.6( 90.6)

FS refers to the fairshare factor. For both jobs, the user's usage is above the target usage, making the fairshare factor negative.

The exact weight given to each factor is subject to change as we attempt to find a balance between high utilization and fairness to all users.

The diagnose command will also tell you how much you and other users have been using the cluster over the past week.

$ diagnose -f
FairShare Information

Depth: 7 intervals Interval Length: 1:00:00:00 Decay Rate: 0.75

FS Policy: UTILIZEDPS
System FS Settings: Target Usage: 0.00 Flags: 0

FSInterval % Target 0 1 2 3 4 5 6
FSWeight ------- ------- 1.0000 0.7500 0.5625 0.4219 0.3164 0.2373 0.1780
TotalUsage 100.00 ------- 23779.0 4358.0 6114.8 10451.7 10853.7 6915.1 7861.7

USER
-------------
bvannemreddy 0.00 5.00 ------- ------- ------- ------- ------- ------- -------
ac.smiyawak* 16.74 5.00 18.66 32.48 0.75 16.23 0.71 17.28 27.11
bernard 0.09 5.00 ------- ------- ------- 0.80 0.00 0.02 0.00
ac.abarsan 2.17 5.00 2.08 ------- ------- ------- ------- 17.27 8.36

The asterisk indicates that the user is above their fairshare target, and that their jobs will be given a negative fairshare weight.

=== More qsub options ===

==== Receiving mail from Torque ====

You can have Torque automatically email you when a job begins, ends, or aborts.

qsub -m <mail-options>

The <mail-options> argument consists of either the single character 'n', or one or more of 'a', 'b', 'e'.

n No mail will be sent.
a Mail will be sent when the job aborts.
b Mail will be sent when the job begins.
e Mail will be sent when the job ends.

==== Redirecting job output ====

Torque will write the standard output of your jobscript to <jobname>.o<jobnumber>, and the standard error to <jobname>.e<jobnumber>. To write them both to the standard output file, submit the job with

qsub -joe ...

or in your jobscript use

#PBS -joe

To write standard output to a different file, submit the job with

qsub -o <some-file>

or in your jobscript:

#PBS -o <some-file>

These can be combined; so, for example, submitting a job with

qsub -joe -o logfile

will cause both standard output and standard error to be written to the file 'logfile' in the directory in which the job is running.

You can see the output file using the command qpeek:

qpeek <jobid>

qpeek has several useful options:

$ qpeek -?
qpeek: Peek into a job's output spool files

Usage: qpeek [options] JOBID

Options:
-c Show all of the output file ("cat", default)
-h Show only the beginning of the output file ("head")
-t Show only the end of the output file ("tail")
-f Show only the end of the file and keep listening ("tail -f")
-# Show only # lines of output
-e Show the stderr file of the job
-o Show the stdout file of the job (default)
-? Display this help message

==== Using fewer than 8 cores per node on Phoenix and fewer than 32 cores per node on Zephyr====

'''For Phoenix'''

* To request and use fewer than 8 cores per node, specify the number of cores you need with the '''ppn''' node property:

qsub -q quadcore -l nodes=<number of nodes>:ppn=<cores per node> ...

For example, to use 4 cores on each of 8 nodes, do

qsub -q quadcore -l nodes=8:ppn=4 ...

* By default, the scheduler will assign multiple jobs to a node as long as the node has resources available, and as long as the jobs are owned by the same user. This behavior can be prevented like this:

qsub -W x=NACCESSPOLICY:SINGLEJOB ...

so the previous command would be

qsub -q quadcore -l nodes=8:ppn=4 -W x=NACCESSPOLICY:SINGLEJOB ...

(It would probably make more sense to have SINGLEJOB as the default, but Maui for some reason does not allow that policy to be overridden.)

'''For Zephyr'''

* To request and use fewer than 32 cores per node, specify the number of cores you need with the '''ppn''' node property:

qsub -q batch -l nodes=<number of nodes>:ppn=<cores per node> ...

For example, to use 4 cores on each of 8 nodes, do

qsub -q batch -l nodes=8:ppn=4 ...

* By default, the scheduler will assign multiple jobs to a node as long as the node has resources available, and as long as the jobs are owned by the same user. This behavior can be prevented like this:

qsub -W x=NACCESSPOLICY:SINGLEJOB ...

so the previous command would be

qsub -q batch -l nodes=8:ppn=4 -W x=NACCESSPOLICY:SINGLEJOB ...

(It would probably make more sense to have SINGLEJOB as the default, but Maui for some reason does not allow that policy to be overridden.)

==== Forcing jobs to run sequentially ====

* To have a job start only after another job has finished, use the '''depend''' job attribute:

qsub -W depend=afterany:<some job-id> ...

For example,

qsub -W depend=afterany:1198 ...

Many other dependency relationships are possible; please see the [http://www.clusterresources.com/torquedocs21/commands/qsub.shtml qsub man page].

==== Passing variables to a jobscript ====

By default, no environment variables are directly exported from the submission environment to the execution environment in which a job runs. This means that you'll usually need to load any modules the job requires in your job script, or in your shell configuration files, as described above, or use one of the following techniques to pass variables to a job.

* To pass all environment variables:

export name=abc; export model=neon
qsub -V ...

* To pass only specified variables:

qsub -v name=abc,model=neon ...

or

export name=abc; export model=neon
qsub -v name,model ...

* You can also specify in a job script which variables Torque should import into the execution environment from the submission environment. To import all environment variables:

#PBS -V

or to import only specified variables:

#PBS -v name,model

Note that you cannot assign values to the variables this way.

=== Job and cluster info ===

* To see a list of jobs which are running, queued, and blocked, use the Maui command showq:

showq [-r | -i | -b ]

For example,

$ showq -r
JobName S Par Effic XFactor Q User Group MHost Procs Remaining StartTime

67174 R DEF 100.00 0.0 pr joe joe n101-ib 1 00:23:32 Wed Apr 28 14:05:12
67233+ R DEF 182.60 1.1 pr jan jan n082-ib 8 14:53:30 Thu Apr 29 13:37:49
66942- R DEF 49.71 0.3 pr lin lin n056-ib 64 18:59:14 Tue Apr 27 08:41:35

The "+" indicates that the job was [[#How_the_scheduler_determines_which_job_to_run_next | backfilled]]; the "—" that the job is [[Using_the_cluster#Quality_of_Service_.28QOS.29 | preemptible]].

showq lists queued (Idle) jobs in descending [[#How_the_scheduler_determines_which_job_to_run_next | priority]] order. showq may show a job as running for up to a minute before it actually starts. To see if it's actually running in that first minute, use Torque's qstat:

qstat -a

* To see how many cores are currently free for a job of a given walltime in each queue, use qbf:

qbf -d <walltime>

* To get more detailed statistics about current cluster usage, use our command qsum:

qsum

* Use pbstop for a graphical display of cluster usage:

pbstop [-?]

* pestat will give node-by-node cluster information:

pestat [-?]

* To get a lot of info from Torque about a running or queued job:

qstat -f <job-id>

* To diagnose problems with a job, you'll need to use both the Torque command tracejob and the Maui command checkjob, and possibly the Maui command diagnose:

tracejob <job-id>
checkjob [-v] <job-id>
diagnose -j <job-id>

* If you want to find out when Maui thinks your job will run, use showstart:

showstart <job-id>

$ showstart 56
job 56 requires 32 procs for 1:40:00
Earliest start in 1:32:48 on Wed Nov 25 18:02:37
Earliest completion in 3:12:48 on Wed Nov 25 19:42:37

* If you want to find out how small and how short you need to make a job for it to run immediately in a particular queue, use the Maui command showbf:

showbf [-f <node-feature>] [-n nodecount] [-d walltime]

where <node-feature> is one of quadcore, eightgb, or thirtytwogb. ("bf" is for backfill, the algorithm used to schedule short and small jobs without interfering with higher priority jobs.) Maui unfortunately doesn't take into account all of our scheduling policies, so showbf's core counts may be too high; nor does it know about our queue walltime limits.

* We have our own utility, qbf, which summarizes the output of showbf for each queue:

$ qbf
Queue Cores Time
quadcore 112 5:49:09
quadcore 40 3:00:00:00
thirtytwogb 8 1:00:00:00

This means that a 40 core job of any duration will run immediately, but a job requesting more than 40 cores will run immediately only if it requests less than 5 hours, 49 minutes. qbf is, however, subject to the same caveat as showbf.

* To get a list of the nodes on which jobs are running (use -s to sort the list):

jobnodes [-s] <jobid> ...

$ jobnodes -s 45124 45873
n008,n015

* To get a list of the nodes on which your jobs are running (use -s to sort the list):

mynodes [-s]

$ mynodes -s
n010,n097

* To see if your application is still running on your nodes:

$ pdsh -w $(mynodes) ps

n046: PID TTY TIME CMD
n046: 13056 ? 00:00:00 bash
n046: 13717 ? 00:00:00 bash
n046: 13721 ? 00:00:00 sshd
n046: 13722 ? 00:00:00 ps

For all of these commands, only the numerical portion of the job-id needs to be specified.

==== qsub-stat ====

qsub-stat is available only on Phoenix and requires the ls-dyna/base module to be loaded.

Most of the commands for tracking your jobs and checking cluster status can be accessed via graphical application qsub-stat. Please refer to section on NoMachine to learn how to run graphical sessions on the TRACC Cluster. qsub-stat has three tabs. The first one shows percentage usage of the two queues (8GB and 32GB) and the disk space usage. Once you move mouse cursor on the meter you will get more detailed information on usage of the queues and the disk space.

[[Image:600px-Resources2.png|thumb|none|left|600px|alt=Resources tab|Resources tab]]

In the Terminal tab majority of explained previously statistics commands can be called through mouse menu. Press and hold right or left mouse button to have access to the commands' list.

[[Image:600px-Terminal2.png|thumb|none|left|600px|alt=Terminal tab|Terminal tab]]

The available commands are:

-------------------------------------------
- Cluster Usage (showq):
- showq
- showq - r (running)
- showq - b (blocked)
- Cluster Usage (qsum)
- Cluster Usage (qstat)
- Available Resources (qbf)
-------------------------------------------
- LS-DYNA Usage (qdyna)
- Madymo Usage (license server report)
- Fluent Usage (license server report)
-------------------------------------------
- Current Login Node activity (htop)
- Node activity (pbstop)
- Node activity (pestat)
-------------------------------------------
- See My Jobs
-------------------------------------------

Current usage of the cluster can also be viewed in the Job Status tab. When you press and hold right or left mouse button on a job, you will see a menu of commands that you can apply to it (Some of them will work only on your own jobs). The available commands are:

-------------------------------------------
- Show additional info:
- qstat -f
- chcekjob
- tracejob
- diagnosejob -j
- showstart
-------------------------------------------
- open standard output
- statistics on master node (htop)
- ssh to master node
-------------------------------------------
- open work directory in browser
- open work directory in terminal
-------------------------------------------
- delete job (qdel)
-------------------------------------------

[[Image:600px-Job_status2.png|thumb|none|left|600px|alt=Job Status tab|Job Status tab]]

Htop command shows live statistics on the master node of the current job.
[[Image:600px-Htop.png|thumb|none|left|600px|alt=Htop|Htop]]

=== Deleting a job ===

To delete a job, use Torque's qdel or Maui's canceljob, which simply calls qdel itself:

qdel <job-id>
canceljob <job-id>

Torque will delete the job immmediately with either command, but Maui may not know about the deletion for up to a minute, during which time it will continue to show up in the output of showq.

If either Torque or Maui tells you it can't delete the job, please send us an email and we'll delete it for you.

Only the numerical portion of the job-id needs to be specified.

=== Interactive Jobs ===

You can get interactive access to a compute node with Torque by using the -I switch to qsub:

$ qsub -I ...
qsub: waiting for job nnnn.host1 to start
qsub: job nnnn.host1 ready
$ cd $PBS_O_WORKDIR

If you want to run a GUI application on the compute node, use the -X switch:

qsub -I -X ...

For this to work, you will need to either use the [[#Graphical_Applications_using_NoMachine_NX | NX client]] to connect to the cluster, or enable trusted [[#Graphical_Applications_using_X | X11]] forwarding when you ssh to the cluster. Windows SSH clients will allow you to enable forwarding in the preferences for the connection. Under Linux and MacOS, you would do this:

$ ssh -Y login.tracc.anl.gov

Also see [[#Accessing_a_compute_node_directly | below]] for an alternative means of directly accessing computes nodes.

=== Using GUI applications on a compute node ===

If you want to use a GUI application on a compute node, you will need to enable trusted [[#Graphical_Applications_using_X | X11]] forwarding when you ssh to the cluster, and again when you ssh to the compute node (as described above). Windows SSH clients will allow you to enable forwarding in the preferences for the connection. Under Linux and MacOS, you would do this:

$ ssh -Y login.tracc.anl.gov

After logging in to a login node, you would then do this to access the compute node:

$ ssh -Y n001

If you're using the [[#Graphical_Applications_using_NoMachine_NX | NX client]] to connect to the cluster, you won't need to ssh to a login node, but you will still need to ssh to a compute node.

=== Accessing a compute node directly ===

The easiest way to do this is by having Torque give you [[#Interactive_jobs | interactive]] access to a node. But if you want to start a process running on a compute node and have it continue to run after you log out, use the method described here.

* You can access any compute node which is running one of your jobs with the command

ssh <compute-node>

You can use the command

qstat -n1 <job-id>

to find out which nodes are running <job-id>.

* If you want Torque to assign you nodes, but don't need it to run a job on your behalf, you can do (for example)

echo sleep 24h | qsub -l select=1,walltime=24:00:00

and then ssh to the node.

=== Using licensed software ===

For some of our software we have a limited number of licenses. To ensure that there will be sufficient licenses available when Maui schedules your job, you will need to submit your job under an application-specific account. Any application wrapper scripts we have will take care of this for you, but if you need to specify the account manually, you would do it like this:

* LS-DYNA:

qsub -A dyna ...

=== Debugging a job ===

Some things to keep in mind or to try when your job doesn't run as expected:

* Jobs don't inherit any environment variables from the submission environment. This means that if your job requires that PATH, LD_LIBRARY_PATH, MPI_HOME, or any other variable be set, you'll need to set it either in the job script, either directly or with a module, or in your shell configuration files.

* Try running your job interactively on a compute node, like this:

$ qsub -I ...
qsub: waiting for job nnnn.host1 to start
qsub: job nnnn.host1 ready
$ cd $PBS_O_WORKDIR

Now run your job script or start up a debugger with your program, just as you would on a login node.

* Try some simple jobs, and look at the standard output and standard error files (STDIN.o<job-id> and STDIN.e<job-id>):

echo | qsub
echo env | qsub

* If you want to see the output of your job as it's being produced, use qpeek:

qpeek <jobid>

=== Undeliverable job output ===

If Torque cannot copy the files containing the output of your job to the job's working directory, the files will eventually be copied to /mnt/gpfs/undelivered. They will be deleted after 10 days.

=== Requeuing of jobs ===

If your job dies due to a problem with the cluster, it will not automatically be requeued. If you would like it to be, do

qsub -r y ...

=== Reserving the cluster ===

Users cannot directly reserve the cluster, but if you think you might have a need for a reservation, please contact us.
[mailto:TRACC_Director@anl.gov Click to send email] or via telephone: 630-252-8224 as far in advance as possible.

Compiling Your Code

2025-04-14T21:50:50Z

Amiot:

if you need help to compile your code, please contact us by [mailto:TRACC_Director@anl.gov clicking to send email].

Backups

2025-04-14T21:50:24Z

Amiot:

We do not operate traditional backup hardware at TRACC. This is mostly because tape backups (or even traditional hard drive backups) would take weeks to restore onto our primary Lustre file system, which has a capacity of 1000 TB and may be expanded in the future as needed. To be able to recover much more efficiently from a major disaster, we are operating a secondary standby Lustre file system of of the same size (also potential subject to future expansion). New Lustre features allow us to create a snapshot of the primary file system at regular intervals (e.g. once or twice per week). The data on the snapshot is then synchronized to the standby Lustre file system over the course of a day or two, ensuring that this secondary copy of all user data is perfectly consistent as of a specific time when the original snapshot was taken.

The snapshots on the primary file system stay around for a limited time period, and can be potentially used if recent user data needs to be recovered (after a user accidentally deletes important files). The snapshots will not allow recovery if the file system encounters a fatal error. This is highly unlikely due to triple redundancy on the Lustre file systems. 3 out of 12 disks in each of the storage groups are allowed to fail without causing data loss, and failing disks are a normal operating condition and are simply replaced and rebuilt on the fly during normal operations (this takes at most a day and users will not notice this). We may shut down file system operations when 2 disks are failing at the same time to have an extra margin of safety while rebuilding the underlying storage. This should be a very rare event and has not been encountered in the past 15 years of operating similar systems. But data integrity is of utmost importance for our operations.

Should we encounter a fatal error with the primary Lustre file system despite all the above mentioned protective measures, we can simply move the standby file system into production (losing possibly a few days' worth of user data since the last synchronization). The standby system can then be replicated onto a fresh primary Lustre file system, with the file systems being swapped out again after a few weeks when this rebuild synchronization is finalized. We do not expect this to ever happen, but we need to be prepared for speedy disaster recovery due to the high value of the data stored on our systems.

Currently, the standby file system is operating in the same physical space as the cluster itself. We are considering to place the standby file system physically into a remote location (very likely the Argonne Enterprise Data Center) to furthermore decrease the potential for catastrophic data loss. The feasibility for sufficient network bandwidth will be evaluated first.

Files & Storage

2025-04-14T21:49:53Z

Amiot:

* Each node in the cluster provides access to local disk storage and to a globally accessible filesystem mounted as /mnt/lustre on ARROW.

* Your home directory, the directory you are placed in when you log in, is /mnt/lustre/home/<user> on ARROW. Your home directory is accessible from all the login nodes and compute nodes. You should use this space for storing files you want to keep long term, such as source code, scripts, input data sets, etc.

=== Collaborative groups ===

A collaborative group can be set up on the TRACC cluster as an easy way for the group to create and manage a shared directory tree of files and programs. A TRACC systems administrator will create the base directory for the directory structure shared by the group. The group base directory will have permissions set to populate the group file structure with directories and files that can be read by all group members, yet still retain some security against accidental modification or deletion of files not owned by a group member working in a group directory.

An example base group directory has the following permissions set:

drwsrws--T 3 joe-user joes-group 32768 Oct 17 17:28 tfhrc

Note that the setuid and setgid flags, “s,” are in the directory permissions in the position where the execute, “x,” flag would normally appear, allowing the directory owner and group members to list (“ls”) the directory and change to the directory (“cd”). The setuid flag is ignored by Linux. The setgid flag causes new files and subdirectories created within the directory to inherit its group ID rather than the group ID of the user who created the file or subdirectory. This inheritance feature allows users, without first changing their group ID with the “newgrp” command, to create files and subdirectories that have group ID of the parent directory instead of the group ID of the user.

In short, the setgid flag allows group members to work in the group directory without doing anything special when creating files; they will be automatically readable by other group members.

The group directory also has the sticky bit set, indicated by a capitol T in the others execute flag position. When the sticky bit is set on a directory, and the group write permission flag is set, group members may create files in the directory, including new subdirectories, and modify and delete the files that they own, but they will not be able to change or delete the files owned by other group members.

In short, the sticky bit set on a group directory, allows users to work safely in collaboration in a group directory: they can create and change their own files, read and copy the files of others, but they cannot change or delete the files of other group members.

When a user creates a subdirectory in the group directory, it inherits the group ID, but the permissions will be assigned based on the user’s mask (umask). The default umask on the TRACC cluster sets read and execute permissions for group members and no permissions for others. It does not set the group write permission. When a group member creates a subdirectory in the group directory, other group members will be able to change into that directory, read the files in the directory, copy the files into another directory, but they will not be able to create files in a new subdirectory that they do not own, unless the owner sets the group write permission for the directory. In that case, it is good practice for the directory owner to also set the sticky bit on the directory to allow other group members to add files to the directory, but prevent them from accidentally changing or deleting files belonging to other group members (i.e. files that they do not own).

In short, to allow other group members to create and modify their own files in a new group subdirectory, but not the files of others do:

chmod g+w subdirectory.name

chmod +t subdirectory.name

Group membership and the group directory provide a relatively easy way for a collaborative group to share data and programs.

Login Nodes Versus Compute Nodes

2025-04-14T21:49:06Z

Amiot:

The cluster consists of login nodes and compute nodes, along with a few other nodes for administration and serving files. The login nodes are shared, and at any time a login node might be in use by 10 or 20 users. For that reason, CPU- and memory-intensive jobs, as well any code that runs a risk of crashing a machine, should be run only on the compute nodes.

Compute nodes can be used both for batch jobs and for interactive jobs. See below for how to use a node [[Running Your Code#Interactive Jobs | interactively]], and how to run [[Running Your Code#Using GUI applications on a compute node | GUI applications]] on a node.

The login nodes are named

login1, login2, login3, and login4

A description of the compute node queues can be [[ARROW Cluster#ARROW Queues | found here.]]

Setting Up Your Environment

2025-04-14T21:45:28Z

Amiot:

=== Shells ===

* Your account will be set up with the bash shell, unless you asked for a different shell when you requested your account. You can find out your current shell with the command <code>echo $SHELL</code>.

* If you'd like a different shell, please contact us. [mailto:TRACC_Director@anl.gov Click to send email].

* The commands <code>chsh</code> and <code>ypchsh</code> don't work on the cluster, and have been disabled.

=== Accessing application software with modules ===

* The cluster uses the [http://modules.sourceforge.net Environment Modules] package to manage access to application software.

* modules documentation is available with the command <code>man module</code>.

* To set up your environment so that an application can be used, use the command

module load <module>

at the command line or in your shell's configuration files.

* You can undo the effect of a module load with

module unload <module>

* It's generally not a good idea to load modules for more than one version of the same application. Instead, swap the modules:

module swap some-application/stable some-application/beta

* Your .bashrc or .cshrc contains sample commands for loading the most common modules. Please see the instructions in those files, or in /etc/skel/.bashrc or /etc/skel/.cshrc, which will be the latest versions of the default configuration files.

* Most modules are one of these two forms:

<application type>/<application>/<version>
<application>/<version>

for example

compiler/pathscale/3.1-147
starccm/15.06.008-R8

Omitting the version number will load the latest stable version of the module.

To see what modules are currently loaded in your environment, do

module list

To see what modules are available, do

module avail

* Modules with "-alpha" or "-beta" appended to the name may not work, either because the software or module is broken.

* To use modules in shell scripts, including Torque jobs scripts, see [[Running Your Code#Job_scripts| below]].

=== Changing ARROW Computing Queues ===
There are currently several computing queues that are available to schedule and some of these queues can only be scheduled by one scheduler (Torque) and the others by a second scheduler (PBS). The nodes scheduled by the queues vary from queue to queue. Be aware that node characteristics (RAM, number of processors, etc) vary from queue to queue. The Torque scheduler is available by default; the PBS scheduler is only available by loading its environmental variable:

The queues "PBS Computing Queues" are available after the environmental specifying the PBS Scheduler is initiated

<pre>
module load pbs/pbs
</pre>
You can review the queues that are available and which scheduler to use to [[ARROW Cluster under construction |'''schedule them by clicking here.''']]

=== File Permissions ===

By default, you will be in your own Unix group, and your files will be readable and writable only by you (that is, the default umask is 027). If you want files to be readable by everyone working on your project, you will need to

* Have your project leader request that we create a Unix group for the project. They also need to let us know who should be in the group.

* Change the group ownership of the files you want to be group-readable:

chgrp <project-group> <filename>

* Make the file group-readable, and, if user-executable, group-executable:

chmod g+rX <filename>

If you want all your new files to be readable by everyone in your group, you'll need to

* Ask us to change your default group. To contact us
[mailto:TRACC_Director@anl.gov click to send email].

Note that a change in your primary group will not affect pre-existing files. To change the group ownership of all your files, do this:

chgrp -R <new-group> ~
chmod -R g+rX ~
chgrp -R $USER ~/.ssh
chmod -R g-rx ~/.ssh

The last two commands are necessary for SSH security; you will lock yourself out of the cluster if you omit them.

Transferring Files

2025-04-14T21:40:35Z

Amiot:

* Files can be transferred to and from the cluster using a client that supports the SFTP (SSH File Transfer Protocol) or with scp (Secure Copy Protocol). Free SFTP clients are available from https://www.putty.org for Windows, and are included with or available for MacOS, and most varieties of Linux/UNIX. Be sure your client is using port 22.

*Popular file transfer clients are WinSCP and FileZilla. You can download WinSCP from http://winscp.net and FileZilla from http://filezilla-project.org.

* Some users have also reported success with [http://www.itefix.no/i2/cwrsync cwrsync], a prepackaged version of Cgywin and rsync.

=== Windows text files ===

Note that text files created under Windows may cause problems on the cluster. This is because Windows and Linux have different conventions for representing a [http://en.wikipedia.org/wiki/Newline newline] in ASCII text files. Some Windows text editors will allow you to save a file in the UNIX/Linux format before transferring the file to the cluster. Alternatively, you can use the Linux command dos2unix to remove the CR characters that Windows uses at the ends of lines:

dos2unix somefile

dos2unix will have no effect on a Linux text file, so it's safe to use on any text file.

You can determine whether a file is in Windows or Linux format with the command

cat -v somefile

Windows text files will have a ^M at the end of each line.

Becoming A User

2025-04-14T21:39:01Z

Amiot:

The TRACC facility, applications software, and resources are limited to users from sponsoring organizations. TRACC welcomes discussions with potential researchers to see whether resources can be provided through existing projects.

== Embargoed or State Sponsors of Terrorism (T-4 Countries) ==

A T-4 country national is anybody who was born in or is a citizen of a country on the Department of State list of "State Sponsors of Terrorism." T-4 countries currently are Cuba, Iran, North Korea, Sudan, and Syria.

A request for TRACC computer access by a T-4 country national requires specific approval by the U.S. Under Secretary of Science. The process requires extensive paperwork that must be submitted for approval. Submittal lead time is 3 or more months with no guarantee the request will be approved.

Principal Investigators are required to inform TRACC of any of their users who are foreign nationals and their countries of citizenship. No work may be performed on TRACC resources on behalf of foreign nationals from sensitive countries.

Main Page

2025-04-14T21:36:37Z

Amiot:

__NOTOC__
{| style="width:100%; background:#7CA0C4; color:white" border="0" cellpadding="0" cellspacing="0"
| align="left" width="99%"|
<blockquote>The Transportation Research and Analysis Computing Center (TRACC)</blockquote>

<blockquote>
This TRACC wiki is an EXTERNAL Argonne collaboration site to support information exchange and collaboration among TRACC users, collaborators, and partners. The purpose of this Wiki is to:
* Document the setup, operation, and FAQs about TRACC supported software and hardware resources.
* Maintain a repository for hardware and software improvements and modifications.
* Share TRACC technical staff, collaborator and user generated "How-To" procedures with the transportation research and development community.
| style="width:1%" | [[Image:Best.JPG]]
|}

==Introduction==
:* [[introduction | '''Introduction to the ARROW Cluster''']] 
:* [[ARROW Cluster#ARROW Queues | '''ARROW Computing Queues''']] 
:* [[Becoming A User| '''Becoming A User''']] 

==[[Using the Cluster| '''Using the Cluster''']]==
:* [[How to Connect| '''How to Connect''']] 
:* [[Transferring Files| '''Transferring Files''']] 
:* [[Setting Up Your Environment| '''Setting Up Your Environment''']] 
:* [[Login Nodes Versus Compute Nodes| '''Login Nodes Versus Compute Nodes''']] 
:* [[Files & Storage| '''Files & Storage''']] 
:* [[Backups| '''Backups''']] 
:* [[Compiling Your Code| '''Compiling Your Code''']] 
:* [[Running Your Code| '''Running Your Code''']] 
:* [[Graphical Applications| '''Graphical Applications''']] 

==TRACC Cluster Software==
:* [[TRACC Cluster Software#Supported On ARROW| '''Supported On ARROW''']] 
:* [[TRACC Cluster Software#Under Investigation| '''Under Investigation''']] 

==TRACC Policies==

:* [[TRACC Policies#Application Restrictions| '''Application Restrictions''']] 
:* [[TRACC Policies#Data Retention| '''Data Retention ''']] 
:* [[TRACC Policies#Email| '''Email ''']] 
:* [[TRACC Policies#Login Nodes| '''Login Nodes''']] 

{{Template:Standard Footer}}

Main Page

2025-04-14T21:35:03Z

Amiot: Removed redirect to Under Construction

Main Page

2025-04-10T21:24:49Z

Amiot: Redirected page to Under Construction

#REDIRECT [[Under Construction]]

Main Page

2025-04-09T20:20:55Z

Amiot: Blanked the page

Main Page

2025-04-09T19:02:37Z

Amiot: Amiot moved page Main Page to Main PageTR

#REDIRECT [[Main PageTR]]

How to Connect

2025-04-08T20:10:37Z

Amiot: /* Access restricted to authorized hosts */

You need an SSH client to connect to Arrow. No other connection protocols are supported. There are 4 general purpose login nodes that you can connect to:

*login1.arrow.tracc.anl.gov
*login2.arrow.tracc.anl.gov
*login3.arrow.tracc.anl.gov
*login4.arrow.tracc.anl.gov

When connecting, please pick a random entry from the above list to evenly distribute users over multiple machines.

For users that work at Argonne or that have an actual Argonne ANL Domain account, you use your assigned account name and password to connect to these login nodes. Just use the ANL Domain user name, without the ANL prefix.

For external collaborators, the bulk of our users, you want to use your external collaborator account that was assigned to you when you applied for an account. These accounts all start with the letters "ac" followed by a period, and then followed by a specific account name. The entire account name, including the leading "ac." portion must be used to connect.

Passwords expire on a regular basis (every 6 months or so), and you will need to update your password with Argonne Account Services. You can call 630-252-9999 if you need any help or detailed instructions.

You will not be able to connect until we have your public IP address on file in our system. Your account will be linked to a few machines with known IP addresses, and if you need to connect from another machine, please contact us to add that machine to your account. Details can be found below.

== How to Connect: Details ==

=== SSH ===

* Access to the TRACC cluster requires use of an SSH client. Free SSH clients are available for Windows, are included or available for MacOS, and all varieties of Linux or UNIX. SSH clients can be terminal applications to work with command lines (e.g. Putty on Windows), or they can be file transfer utilities (WinSCP, FileZilla). In any case, cluster connections are all using the SSH protocol (including sftp).

* If you're using a simple graphical application on the cluster through an SSH connection and you are either connecting from a Mac or Linux desktop machine or you have an X server installed on your Windows machine, you may want to enable X11 forwarding to your client. This is a method that uses strictly the old X protocol, and performance is very limited (due to the SSH encryption overhead). You should consider using one of the remote GUI options listed below instead if you can. If you want to keep it simple though, you can do this (using typically a corresponding check box in Windows clients):

ssh -X username@login1.arrow.tracc.anl.gov'''
'''--[[User:Amiot|Amiot]] ([[User talk:Amiot|talk]]) 15:28, January 14, 2025 (CST)
* Alternatively, client software like the [[Graphical Applications#GRAPHICAL APPLICATIONS Using NoMachine NX | NoMachine NX client]] can be used to connect, providing a full graphical user interface. This requires the installation of the free NoMachine client software.

* Another alternative is the [[Graphical Applications#GRAPHICAL APPLICATIONS Using NoMachine NX | free X2Go client]], which provides a near identical user experience. Details can be found [[Graphical Applications#GRAPHICAL APPLICATIONS Using NoMachine NX | here]].

=== Access restricted to authorized hosts ===

* Access to the cluster is currently restricted to pre-approved hosts. Please contact us [mailto:TRACC_Director@anl.gov clicking to send email] with the hostnames of the systems from which you would like to access the cluster. Argonne users with Argonne Domain accounts can connect through Argonne's VPN service; please contact us by [mailto:TRACC_Director@anl.gov clicking to send email] for details.

* Under Windows, you can find out your host name with the command "ipconfig". Note that IP addresses beginning with "10." or "192.168." are private addresses assigned by your home router, and those are not accessible from outside your home network. If you have such an address, you can usually find your public IP address at (for example) http://whatismyip.com.

Setting Up Your Environment

2025-03-11T20:33:50Z

Amiot: /* File permissions */

=== Shells ===

* Your account will be set up with the bash shell, unless you asked for a different shell when you requested your account. You can find out your current shell with the command <code>echo $SHELL</code>.

* If you'd like a different shell, please contact us. [mailto:TRACC_Director@anl.gov Click to send email].

* The commands <code>chsh</code> and <code>ypchsh</code> don't work on the cluster, and have been disabled.

=== Accessing application software with modules ===

* The cluster uses the [http://modules.sourceforge.net Environment Modules] package to manage access to application software.

* modules documentation is available with the command <code>man module</code>.

* To set up your environment so that an application can be used, use the command

module load <module>

at the command line or in your shell's configuration files.

* You can undo the effect of a module load with

module unload <module>

* It's generally not a good idea to load modules for more than one version of the same application. Instead, swap the modules:

module swap some-application/stable some-application/beta

* Your .bashrc or .cshrc contains sample commands for loading the most common modules. Please see the instructions in those files, or in /etc/skel/.bashrc or /etc/skel/.cshrc, which will be the latest versions of the default configuration files.

* Most modules are one of these two forms:

<application type>/<application>/<version>
<application>/<version>

for example

compiler/pathscale/3.1-147
starccm/15.06.008-R8

Omitting the version number will load the latest stable version of the module.

To see what modules are currently loaded in your environment, do

module list

To see what modules are available, do

module avail

* Modules with "-alpha" or "-beta" appended to the name may not work, either because the software or module is broken.

* To use modules in shell scripts, including Torque jobs scripts, see [[Running Your Code#Job_scripts| below]].

=== Changing ARROW Computing Queues ===
There are currently several computing queues that are available to schedule and some of these queues can only be scheduled by one scheduler (Torque) and the others by a second scheduler (PBS). The nodes scheduled by the queues vary from queue to queue. Be aware that node characteristics (RAM, number of processors, etc) vary from queue to queue. The Torque scheduler is available by default; the PBS scheduler is only available by loading its environmental variable:

The queues "PBS Computing Queues" are available after the environmental specifying the PBS Scheduler is initiated

<pre>
module load pbs/pbs
</pre>
You can review the queues that are available and which scheduler to use to [[ARROW Cluster under construction |'''schedule them by clicking here.''']]

=== File Permissions ===

By default, you will be in your own Unix group, and your files will be readable and writable only by you (that is, the default umask is 027). If you want files to be readable by everyone working on your project, you will need to

* Have your project leader request that we create a Unix group for the project. They also need to let us know who should be in the group.

* Change the group ownership of the files you want to be group-readable:

chgrp <project-group> <filename>

* Make the file group-readable, and, if user-executable, group-executable:

chmod g+rX <filename>

If you want all your new files to be readable by everyone in your group, you'll need to

* Ask us to change your default group. To contact us
[mailto:TRACC_Director@anl.gov click to send email].

Note that a change in your primary group will not affect pre-existing files. To change the group ownership of all your files, do this:

chgrp -R <new-group> ~
chmod -R g+rX ~
chgrp -R $USER ~/.ssh
chmod -R g-rx ~/.ssh

The last two commands are necessary for SSH security; you will lock yourself out of the cluster if you omit them.

:::::::::::::::[[New main|'''Return To Main Page''']]

Setting Up Your Environment

2025-03-11T20:30:57Z

Amiot: /* File Permissions */

=== Shells ===

* Your account will be set up with the bash shell, unless you asked for a different shell when you requested your account. You can find out your current shell with the command <code>echo $SHELL</code>.

* If you'd like a different shell, please contact us. [mailto:TRACC_Director@anl.gov Click to send email].

* The commands <code>chsh</code> and <code>ypchsh</code> don't work on the cluster, and have been disabled.

=== Accessing application software with modules ===

* The cluster uses the [http://modules.sourceforge.net Environment Modules] package to manage access to application software.

* modules documentation is available with the command <code>man module</code>.

* To set up your environment so that an application can be used, use the command

module load <module>

at the command line or in your shell's configuration files.

* You can undo the effect of a module load with

module unload <module>

* It's generally not a good idea to load modules for more than one version of the same application. Instead, swap the modules:

module swap some-application/stable some-application/beta

* Your .bashrc or .cshrc contains sample commands for loading the most common modules. Please see the instructions in those files, or in /etc/skel/.bashrc or /etc/skel/.cshrc, which will be the latest versions of the default configuration files.

* Most modules are one of these two forms:

<application type>/<application>/<version>
<application>/<version>

for example

compiler/pathscale/3.1-147
starccm/15.06.008-R8

Omitting the version number will load the latest stable version of the module.

To see what modules are currently loaded in your environment, do

module list

To see what modules are available, do

module avail

* Modules with "-alpha" or "-beta" appended to the name may not work, either because the software or module is broken.

* To use modules in shell scripts, including Torque jobs scripts, see [[Running Your Code#Job_scripts| below]].

=== Changing ARROW Computing Queues ===
There are currently several computing queues that are available to schedule and some of these queues can only be scheduled by one scheduler (Torque) and the others by a second scheduler (PBS). The nodes scheduled by the queues vary from queue to queue. Be aware that node characteristics (RAM, number of processors, etc) vary from queue to queue. The Torque scheduler is available by default; the PBS scheduler is only available by loading its environmental variable:

The queues "PBS Computing Queues" are available after the environmental specifying the PBS Scheduler is initiated

<pre>
module load pbs/pbs
</pre>
You can review the queues that are available and which scheduler to use to [[ARROW Cluster under construction |'''schedule them by clicking here.''']]

=== File permissions ===

By default, you will be in your own Unix group, and your files will be readable and writable only by you (that is, the default umask is 027). If you want files to be readable by everyone working on your project, you will need to

* Have your project leader request that we create a Unix group for the project. They also need to let us know who should be in the group.

* Change the group ownership of the files you want to be group-readable:

chgrp <project-group> <filename>

* Make the file group-readable, and, if user-executable, group-executable:

chmod g+rX <filename>

If you want all your new files to be readable by everyone in your group, you'll need to

* Ask us to change your default group. To contact us
[mailto:TRACC_Director@anl.gov click to send email].

Note that a change in your primary group will not affect pre-existing files. To change the group ownership of all your files, do this:

chgrp -R <new-group> ~
chmod -R g+rX ~
chgrp -R $USER ~/.ssh
chmod -R g-rx ~/.ssh

The last two commands are necessary for SSH security; you will lock yourself out of the cluster if you omit them.

:::::::::::::::[[New main|'''Return To Main Page''']]

Setting Up Your Environment

2025-03-11T20:27:06Z

Amiot: /* File permissions */

=== Shells ===

* Your account will be set up with the bash shell, unless you asked for a different shell when you requested your account. You can find out your current shell with the command <code>echo $SHELL</code>.

* If you'd like a different shell, please contact us. [mailto:TRACC_Director@anl.gov Click to send email].

* The commands <code>chsh</code> and <code>ypchsh</code> don't work on the cluster, and have been disabled.

=== Accessing application software with modules ===

* The cluster uses the [http://modules.sourceforge.net Environment Modules] package to manage access to application software.

* modules documentation is available with the command <code>man module</code>.

* To set up your environment so that an application can be used, use the command

module load <module>

at the command line or in your shell's configuration files.

* You can undo the effect of a module load with

module unload <module>

* It's generally not a good idea to load modules for more than one version of the same application. Instead, swap the modules:

module swap some-application/stable some-application/beta

* Your .bashrc or .cshrc contains sample commands for loading the most common modules. Please see the instructions in those files, or in /etc/skel/.bashrc or /etc/skel/.cshrc, which will be the latest versions of the default configuration files.

* Most modules are one of these two forms:

<application type>/<application>/<version>
<application>/<version>

for example

compiler/pathscale/3.1-147
starccm/15.06.008-R8

Omitting the version number will load the latest stable version of the module.

To see what modules are currently loaded in your environment, do

module list

To see what modules are available, do

module avail

* Modules with "-alpha" or "-beta" appended to the name may not work, either because the software or module is broken.

* To use modules in shell scripts, including Torque jobs scripts, see [[Running Your Code#Job_scripts| below]].

=== Changing ARROW Computing Queues ===
There are currently several computing queues that are available to schedule and some of these queues can only be scheduled by one scheduler (Torque) and the others by a second scheduler (PBS). The nodes scheduled by the queues vary from queue to queue. Be aware that node characteristics (RAM, number of processors, etc) vary from queue to queue. The Torque scheduler is available by default; the PBS scheduler is only available by loading its environmental variable:

The queues "PBS Computing Queues" are available after the environmental specifying the PBS Scheduler is initiated

<pre>
module load pbs/pbs
</pre>
You can review the queues that are available and which scheduler to use to [[ARROW Cluster under construction |'''schedule them by clicking here.''']]

=== File Permissions ===

By default, you will be in your own Unix group, and your files will be readable and writable only by you (that is, the default umask is 027). If you want files to be readable by everyone working on your project, you will need to

* Have your project leader request that we create a Unix group for the project. They also need to let us know who should be in the group.

* Change the group ownership of the files you want to be group-readable:

chgrp <project-group> <filename>

* Make the file group-readable, and, if user-executable, group-executable:

chmod g+rX <filename>

If you want all your new files to be readable by everyone in your group, you'll need to

* Ask us to change your default group. To contact us
[mailto:TRACC_Director@anl.gov click to send email].

Note that a change in your primary group will not affect pre-existing files. To change the group ownership of all your files, do this:

chgrp -R <new-group> ~
chmod -R g+rX ~
chgrp -R $USER ~/.ssh
chmod -R g-rx ~/.ssh

The last two commands are necessary for SSH security; you will lock yourself out of the cluster if you omit them.

:::::::::::::::[[New main|'''Return To Main Page''']]

Setting Up Your Environment

2025-03-11T20:19:12Z

Amiot: /* Changing ARROW Computing Queues */

=== Shells ===

* Your account will be set up with the bash shell, unless you asked for a different shell when you requested your account. You can find out your current shell with the command <code>echo $SHELL</code>.

* If you'd like a different shell, please contact us. [mailto:TRACC_Director@anl.gov Click to send email].

* The commands <code>chsh</code> and <code>ypchsh</code> don't work on the cluster, and have been disabled.

=== Accessing application software with modules ===

* The cluster uses the [http://modules.sourceforge.net Environment Modules] package to manage access to application software.

* modules documentation is available with the command <code>man module</code>.

* To set up your environment so that an application can be used, use the command

module load <module>

at the command line or in your shell's configuration files.

* You can undo the effect of a module load with

module unload <module>

* It's generally not a good idea to load modules for more than one version of the same application. Instead, swap the modules:

module swap some-application/stable some-application/beta

* Your .bashrc or .cshrc contains sample commands for loading the most common modules. Please see the instructions in those files, or in /etc/skel/.bashrc or /etc/skel/.cshrc, which will be the latest versions of the default configuration files.

* Most modules are one of these two forms:

<application type>/<application>/<version>
<application>/<version>

for example

compiler/pathscale/3.1-147
starccm/15.06.008-R8

Omitting the version number will load the latest stable version of the module.

To see what modules are currently loaded in your environment, do

module list

To see what modules are available, do

module avail

* Modules with "-alpha" or "-beta" appended to the name may not work, either because the software or module is broken.

* To use modules in shell scripts, including Torque jobs scripts, see [[Running Your Code#Job_scripts| below]].

=== Changing ARROW Computing Queues ===
There are currently several computing queues that are available to schedule and some of these queues can only be scheduled by one scheduler (Torque) and the others by a second scheduler (PBS). The nodes scheduled by the queues vary from queue to queue. Be aware that node characteristics (RAM, number of processors, etc) vary from queue to queue. The Torque scheduler is available by default; the PBS scheduler is only available by loading its environmental variable:

The queues "PBS Computing Queues" are available after the environmental specifying the PBS Scheduler is initiated

<pre>
module load pbs/pbs
</pre>
You can review the queues that are available and which scheduler to use to [[ARROW Cluster under construction |'''schedule them by clicking here.''']]

=== File permissions ===

By default, you will be in your own Unix group, and your files will be readable and writable only by you (that is, the default umask is 027). If you want files to be readable by everyone working on your project, you will need to

* Have your project leader request that we create a Unix group for the project. They also need to let us know who should be in the group.

* Change the group ownership of the files you want to be group-readable:

chgrp <project-group> <filename>

* Make the file group-readable, and, if user-executable, group-executable:

chmod g+rX <filename>

If you want all your new files to be readable by everyone in your group, you'll need to

* Ask us to change your default group. To contact us
[mailto:TRACC_Director@anl.gov click to send email].

Note that a change in your primary group will not affect pre-existing files. To change the group ownership of all your files, do this:

chgrp -R <new-group> ~
chmod -R g+rX ~
chgrp -R $USER ~/.ssh
chmod -R g-rx ~/.ssh

The last two commands are necessary for SSH security; you will lock yourself out of the cluster if you omit them.

:::::::::::::::[[New main|'''Return To Main Page''']]

Setting Up Your Environment

2025-03-11T20:17:11Z

Amiot: /* Changing ARROW Computing Queues */

=== Shells ===

* Your account will be set up with the bash shell, unless you asked for a different shell when you requested your account. You can find out your current shell with the command <code>echo $SHELL</code>.

* If you'd like a different shell, please contact us. [mailto:TRACC_Director@anl.gov Click to send email].

* The commands <code>chsh</code> and <code>ypchsh</code> don't work on the cluster, and have been disabled.

=== Accessing application software with modules ===

* The cluster uses the [http://modules.sourceforge.net Environment Modules] package to manage access to application software.

* modules documentation is available with the command <code>man module</code>.

* To set up your environment so that an application can be used, use the command

module load <module>

at the command line or in your shell's configuration files.

* You can undo the effect of a module load with

module unload <module>

* It's generally not a good idea to load modules for more than one version of the same application. Instead, swap the modules:

module swap some-application/stable some-application/beta

* Your .bashrc or .cshrc contains sample commands for loading the most common modules. Please see the instructions in those files, or in /etc/skel/.bashrc or /etc/skel/.cshrc, which will be the latest versions of the default configuration files.

* Most modules are one of these two forms:

<application type>/<application>/<version>
<application>/<version>

for example

compiler/pathscale/3.1-147
starccm/15.06.008-R8

Omitting the version number will load the latest stable version of the module.

To see what modules are currently loaded in your environment, do

module list

To see what modules are available, do

module avail

* Modules with "-alpha" or "-beta" appended to the name may not work, either because the software or module is broken.

* To use modules in shell scripts, including Torque jobs scripts, see [[Running Your Code#Job_scripts| below]].

=== Changing ARROW Computing Queues ===
There are currently several computing queues that are available to schedule and some of these queues can only be scheduled by one scheduler (Torque) and the others by a second scheduler (PBS). The nodes scheduled by the queues vary from queue to queue. Be aware that node characteristics (RAM, number of processors, etc) vary from queue to queue. The Torque scheduler is available by default; the PBS scheduler is only available by loading its environmental variable:

The queues "PBS Computing Queues" are available after the environmental specifying the PBS Scheduler is initiated

<pre>
module load pbs/pbs
</pre>
You can review the queues that are available and which scheduler to use to [[ARROW Cluster under construction |'''schedule them by clicking here.''']]
You can return to the queues that are available and which scheduler to use to [[ARROW Cluster under construction |'''schedule them by clicking here.''']]

=== File permissions ===

By default, you will be in your own Unix group, and your files will be readable and writable only by you (that is, the default umask is 027). If you want files to be readable by everyone working on your project, you will need to

* Have your project leader request that we create a Unix group for the project. They also need to let us know who should be in the group.

* Change the group ownership of the files you want to be group-readable:

chgrp <project-group> <filename>

* Make the file group-readable, and, if user-executable, group-executable:

chmod g+rX <filename>

If you want all your new files to be readable by everyone in your group, you'll need to

* Ask us to change your default group. To contact us
[mailto:TRACC_Director@anl.gov click to send email].

Note that a change in your primary group will not affect pre-existing files. To change the group ownership of all your files, do this:

chgrp -R <new-group> ~
chmod -R g+rX ~
chgrp -R $USER ~/.ssh
chmod -R g-rx ~/.ssh

The last two commands are necessary for SSH security; you will lock yourself out of the cluster if you omit them.

:::::::::::::::[[New main|'''Return To Main Page''']]

Setting Up Your Environment

2025-03-11T20:06:55Z

Amiot: /* File permissions */

=== Shells ===

* Your account will be set up with the bash shell, unless you asked for a different shell when you requested your account. You can find out your current shell with the command <code>echo $SHELL</code>.

* If you'd like a different shell, please contact us. [mailto:TRACC_Director@anl.gov Click to send email].

* The commands <code>chsh</code> and <code>ypchsh</code> don't work on the cluster, and have been disabled.

=== Accessing application software with modules ===

* The cluster uses the [http://modules.sourceforge.net Environment Modules] package to manage access to application software.

* modules documentation is available with the command <code>man module</code>.

* To set up your environment so that an application can be used, use the command

module load <module>

at the command line or in your shell's configuration files.

* You can undo the effect of a module load with

module unload <module>

* It's generally not a good idea to load modules for more than one version of the same application. Instead, swap the modules:

module swap some-application/stable some-application/beta

* Your .bashrc or .cshrc contains sample commands for loading the most common modules. Please see the instructions in those files, or in /etc/skel/.bashrc or /etc/skel/.cshrc, which will be the latest versions of the default configuration files.

* Most modules are one of these two forms:

<application type>/<application>/<version>
<application>/<version>

for example

compiler/pathscale/3.1-147
starccm/15.06.008-R8

Omitting the version number will load the latest stable version of the module.

To see what modules are currently loaded in your environment, do

module list

To see what modules are available, do

module avail

* Modules with "-alpha" or "-beta" appended to the name may not work, either because the software or module is broken.

* To use modules in shell scripts, including Torque jobs scripts, see [[Running Your Code#Job_scripts| below]].

=== Changing ARROW Computing Queues ===
There are currently several computing queues that are available to schedule and some of these queues can only be scheduled by one scheduler (Torque) and the others by a second scheduler (PBS). The nodes scheduled by the queues vary from queue to queue. Be aware that node characteristics (RAM, number of processors, etc) vary from queue to queue. The Torque scheduler is availale by default; the PBS scheduler is only available by loading its environmental variable:

The queues "PBS Computing Queues" are available after the environmental specifying the PBS Scheduler is initiated

<pre>
module load pbs/pbs
</pre>
You can review the queues that are available and which scheduler to use to [[ARROW Cluster under construction |'''schedule them by clicking here.''']]

=== File permissions ===

By default, you will be in your own Unix group, and your files will be readable and writable only by you (that is, the default umask is 027). If you want files to be readable by everyone working on your project, you will need to

* Have your project leader request that we create a Unix group for the project. They also need to let us know who should be in the group.

* Change the group ownership of the files you want to be group-readable:

chgrp <project-group> <filename>

* Make the file group-readable, and, if user-executable, group-executable:

chmod g+rX <filename>

If you want all your new files to be readable by everyone in your group, you'll need to

* Ask us to change your default group. To contact us
[mailto:TRACC_Director@anl.gov click to send email].

Note that a change in your primary group will not affect pre-existing files. To change the group ownership of all your files, do this:

chgrp -R <new-group> ~
chmod -R g+rX ~
chgrp -R $USER ~/.ssh
chmod -R g-rx ~/.ssh

The last two commands are necessary for SSH security; you will lock yourself out of the cluster if you omit them.

:::::::::::::::[[New main|'''Return To Main Page''']]

TRACC Policies

2025-03-09T01:35:20Z

Amiot:

TRACC Cluster Software

2025-03-09T01:33:29Z

Amiot:

Graphical Applications

2025-03-09T01:30:25Z

Amiot:

==General Graphic Application Comments==

This chapter describes the use of graphical user interfaces to bring the Linux desktop to your local machine. This gives you the ability to work with graphical applications, browse and edit files in a graphical manner that is similar to your usual work patterns, and establish remote sessions that stay alive while you disconnect from the cluster. You can of course fully log out when appropriate, but by default, the session stays alive until you reconnect at a later time. This allows you to keep interactive applications running for very long times if needed.

There are three ways to connect to Arrow to run graphic applications. These are:

* Running the NoMachine NX client
* Running the X2Go client
* Running X and using an SSH terminal client to forward the X protocol

Running NoMachine and X2Go are the most efficient because they transfer information between the cluster and the client using data compression and a special protocol to minimize data transmissions.

Note: When using the third option, you will not have a full Linux desktop on your screen. Instead, Linux applications can bring up your user interface on your client for interaction. This is much slower than the other two options, but may be useful for your specific application. In this case, all rendering is done on your client rather than on the remote server.

Let's assume that you have an X server running on your machine, and that you have the ssh command available. In that case, you would do the following:

ssh -X userid@login1.arrow.tracc.anl.gov
enter your password
xeyes

This last command is just for testing. You should see a set of eyes that can be moved around using your mouse. If this works, then your graphical application scan be used remotely.

==Graphical Applications Using NoMachine NX==

To provide for the remote use of graphical applications on the cluster, we've installed [http://www.nomachine.com NoMachine NX] on the login hosts. This method allow you to establish a remote desktop on the login machine, which stays alive even when you disconnect from it. You can of course log out of that connection, but by default, the desktop stays alive, allowing you to keep interactive applications running while disconnected. To use this software,

* download and install the free [http://www.nomachine.com/download.php NX Client].
* connect to the TRACC cluster using one of the login nodes:
** login1.arrow.tracc.anl.gov
** login2.arrow.tracc.anl.gov
** login3.arrow.tracc.anl.gov
** login4.arrow.tracc.anl.gov
* The NX protocol for connections is not supported. Please choose SSH from the pull-down menu.
* When connecting the very first time, you are prompted to confirm that you are connecting to the correct server, showing a server key. Confirm and continue.
* When doing this for the first time, create a new Linux Desktop. This will be shown in the future when connecting again.
* Please remember that you need a proper account, password, and that we need to have your public IP address on file for all of this to work properly.



If you have problems, please contact us by [mailto:TRACC_Director@anl.gov clicking to send email] or via telephone at 630-252-8224.

=== Terminating and Disconnecting ===

When you quit an NX session, you can either terminate it or disconnect from it. Terminating it will close all your running programs. If you disconnect, you can reconnect later to the same session. Interactive applications will continue running while disconnected, so this is a convenient way to occasionally check on progress, or to work with long running applications despite possible network issues.

If you close the NX client window, you will be simply disconnected and can resume this connection at a later time. To fully log out, use the Linux pull-down command at the top left of your screen, and choose the bottom entry to log out.

=== Reconnecting ===

If you want to reconnect to an NX session, you'll have to log in to the same login node on which it was started.

=== Problems ===

We will post trouble-shooting instructions in this section if ever needed. The latest versions are very stable.

==Graphical Applications Using X2Go==

X2Go can be installed by following the [https://wiki.x2go.org/doku.php/doc:installation:x2goclient instructions here.]. Instructions are provided for Windows, OSX, and a number of other client OSs.

After installing the client software, create a new connection with the following settings

* connection to one of our login servers
** login1.arrow.tracc.anl.gov
** login2.arrow.tracc.anl.gov
** login3.arrow.tracc.anl.gov
** login4.arrow.tracc.anl.gov
* account name and password of course
* IMPORTANT: from the pull-down menu, choose the XFCE desktop
* Also, we recommend turning off remote audio (it won't work anyway)

* on Macs: It should be noticed that if you don't already have XQuartz installed on OSX, you will need to do that before installing X2Go.
* A general X2Go tutorial can be [https://wiki.anl.gov/wiki_traccwiki/index.php?title=Graphical_Applications&action=editfound here.]

==Graphical Applications Using X==
[http://sourceforge.net/projects/xming xming] is a convenient and easy way to install an X server for Windows. However, if you're doing pre- or post-processing that requires sending a lot of graphics over the network, we recommend NoMachine NX or X2Go because their compression of the graphics yields a significant speed advantage over X. Also, rendering is generally perform on the server side rather than your local machine, speeding things up. In addition, the X protocol is very chatty, and performs poorly over the SSH encryption channel.

*To install X on a Mac, refer to this [https://princetonuniversity.github.io/PUbootcamp/ssh-instructions site ]
* The Mac uses XQuart to implement X applications and should be installed if not alread installed on your version of OSX. To bring up the window, connect to the cluster using the following (-X or -Y is required):
ssh -Y userid@login1.arrow.tracc.anl.gov

:::::::::::::::[[New main|'''Return To Main Page''']]

Running Your Code

2025-03-09T01:29:34Z

Amiot:

= Careful, this chapter is quite outdated and needs to be rewritten =

== Running Jobs ==

'''Please do not run CPU- or memory-intensive tasks on the login nodes; always run them on the compute nodes. Resource-intensive processes running on login nodes may be killed at any time, without warning or notification.'''

=== Getting started ===

Jobs are submitted to the cluster using Torque and scheduled by Maui. Some documentation of the commands that are used by Torque and Maui can be found on the man pages. For example:
man qsub

To use Torque and Maui, you'll need to make sure the base [[Setting Up Your Environment#Accessing application software with modules | module]] is loaded in your shell's setup files:

module load base

Feel free to to contact us. [mailto:TRACC_Director@anl.gov Click to send email] or via telephone: 630-252-8224 if you need help with Torque or Maui. Note that other documentation you might find regarding Torque and Maui may not apply to the TRACC cluster because of differences in how we've configured the software.

=== Torque's view of the cluster ===

Torque and Maui are both used to schedule and run job for both the arrow and nhtsa queues.

=== Submitting jobs ===

==== Job scripts ====

The usual way of running a job with Torque is to submit a shell script. For most commercial applications we have wrapper scripts to automatically generate job scripts; see our [[TRACC Cluster Software#Supported On ARROW | application notes for our supported software]], or contact your sponsor for more info, then skip ahead to information about [[#qsub usage | qsub]].

A simple script to run an MPI job would be

#!/bin/bash

# PBS directives must precede all executable commands

cd $PBS_O_WORKDIR

## HPMPI
# mpirun -hostfile $PBS_NODEFILE ./mpi-application

## Intel MPI
# mpirun ./mpi-application

If you need to load a module in your shell script, do it like this, for scripts written in /bin/sh or /bin/bash:

. /etc/profile.d/modules.sh
module load <some-module>

or like this, if your script is written in /bin/csh or /bin/tcsh:

. /etc/profile.d/modules.csh
module load <some-module>

or like this, for Python:

import sys
sys.path.append("/soft/modules/current/init")

import envmodule
envmodule.module("load <some-module>")

or like this, for Perl:

use lib "/soft/modules/current/init";
use envmodule;
module("load <some-module");

For other languages, you can usually achieve the same effect by loading the module in your shell's configuration files (.bashrc or .cshrc).

==== Torque variables ====

* Torque sets several variables which you can use in your job scripts. See the qsub man page for the complete list.

* All Torque variables begin with "PBS", as Torque used to be known as OpenPBS (which was the precursor of PBS Pro).

* The most important Torque variables to be aware of are

PBS_O_WORKDIR

which is the working directory at the time of job submission, and

PBS_NODEFILE

which contains a list of the compute nodes which Torque has assigned your job, in the form

n001
n001
n001
n001
n001
n001
n001
n001

with each node listed once for each core.

==== qsub usage ====

The command is used to submit jobs to the cluster. qsub's behavior can be customized with command line options or script directives. Note that all script directives must precede all executable commands, so it's best to put all directives at the top of the script file.

'''For example:'''

* To use qsub to submit a 32-core job; to the batch queue which is the default queue; with a maximum walltime of 10 hours (meaning that the job will be killed after 10 hours have elapsed):

qsub -l nodes=4:ppn=8,walltime=10:00:00 jobscript

This can be specified in the jobscript with

#PBS -l nodes=4:ppn=8
#PBS -l walltime=10:00:00

in which case the job would be submitted with

qsub jobscript

* The default resource request if one isn't specified on the command line or in the jobscript is

nodes=1:ppn=1 -l walltime=5:00

meaning 1 core on 1 node, with a walltime of 5 minutes.

'''For example:'''

* For scientific applications that rely heavily on floating point operations such as STAR-CCM+, LS-DYNA, OpenFOAM, and University CFD research codes, users should specify 16 cores per node ( PPN=16 ), and not 32 since each node has a total of 16 floating point units (FPUs). Specifying 32 cores ( PPN=32 ) per compute node will not provide substantial performance gains, but will use up commerical licenses.
* To use the nhtsa queue which is not the default queue and has 64 GB nodes:
* To use qsub to submit a 64-core job with a maximum walltime of 10 hours (meaning that the job will be killed after 10 hours have elapsed):
'''Steve can you modify the following command to use the nhtsa queue.... Larry'''
qsub -l nodes=2:ppn=32,walltime=10:00:00 jobscript

This can be specified in the jobscript with

#PBS -l nodes=2:ppn=32
#PBS -l walltime=10:00:00

in which case the job would be submitted with

qsub jobscript

* The default resource request if one isn't specified on the command line or in the jobscript is

nodes=1:ppn=1 -l walltime=5:00

meaning 1 core on 1 node, with a walltime of 5 minutes.

==== Initial directory ====

Torque by default will run your script from your home directory. You can specify a different initial directory with

qsub -d ...

For example, to start in your current directory:

qsub -d `pwd`

==== Large memory jobs ====
There are two queues that are available for large memory jobs.

* batch64 queue
** 2 nodes numbered n003 and n004
** Same design as batch queue
** 64GB of RAM per node
** available for general use

* batch128 queue
** 2 nodes numbered n001 and n002
** Same design as batch queue
** 128GB of RAM per node
** available for general use

As an example, to submit your jobs to the batch128 queue:

qsub -q batch128 ...

Jobs on the nodes are limited to 72 hours.

Swap has been disabled on these nodes.

==== Short jobs ====

Weekdays from 7am CDT to 7pm CDT, two nodes are set aside for jobs on Phoenix with walltimes of 2 hours or less. Maui will automatically schedule short jobs to these nodes. No policy is currently in place for Zephyr.

=== Usage policies ===

==== Maximum walltime ====

Jobs which exceed these walltime limits will be rejected by Torque.

On Zephyr:
batch queue: 168 hours
batch64 queue: 72 hours
batch128 queue: 72 hours

On Phoenix:
quadcore queue: 168 hours
thirtytwogb queue: 72 hours

==== Quotas ====

Job which exceed the following quotas will be blocked by Maui. Blocked jobs can be seen with the command

showq -b

==== Node quotas ====

The batch64GB and batch128GB queues on Zephyr and the thirtytwogb queue on Phoenix have a limit of 1 job per user. This can also be overridden for a particular job by making the job preemptible using the Maui [[Using_the_cluster#Quality_of_Service_.28QOS.29 | QOS]] feature.

==== Queue quotas ====

The batch64GB and batch128GB queues on Zephyr and the thirtytwogb queue on Phoenix have a limit of 1 job per user. This can also be overridden for a particular job by making the job preemptible using the Maui [[Using_the_cluster#Quality_of_Service_.28QOS.29 | QOS]] feature.

=== Quality of Service (QOS) ===

In order to override the cluster's usage quotas, you can submit your jobs with a preemptible QOS. The caveat is that a preemptible job may be killed at any time, regardless of its wallclock limit, if doing so would enable another user's ordinary, non-preemptible job to run (your own non-preemptible jobs will not preempt your preemptible jobs).

To make a job preemptible, you can submit it with a QOS (quality of service) flag. For example:

qsub -W x=QOS:preemptible ... <jobscript>

You can make a submitted job (whether queued or running) preemptible with the Maui comand setqos:

setqos preemptible <jobid>

And you can make a job non-preemptible by setting its QOS to DEFAULT:

setqos DEFAULT <jobid>

You can check a job's QOS with the command [[Using_the_cluster#Job_and_cluster_info | checkjob]].

Note that because of a bug in Torque, if you want to specify a [[Using_the_cluster#Using_fewer_than_8_cores_per_node | node access policy]] and a QOS, you should not specify the QOS to Torque, but should instead submit the job without a QOS, then set the QOS using setqos.

=== How the scheduler determines which job to run next ===

Maui runs queued jobs in priority order (blocked jobs are not considered). Priority is a function of

* your recent cluster usage (the less usage, the higher the priority)

* the number of nodes requested (the more nodes, the higher the priority)

* a combination of the walltime requested (the less time, the higher the priority) and how long the job has been waiting to run

This last factor is referred to as the "expansion factor" in the Maui documentation, and is equal to

1 + time since submission / wallclock time

which, all else being equal, is designed to cause jobs to wait on average only as long as their requested walltime.

Maui also uses "backfill" to schedule short and small jobs when doing so would not interfere with higher priority jobs.

To find out how the priority of your job was calculated, use the Maui command '''diagnose''', which will show something like this:

$ diagnose -p
diagnosing job priority information (partition: ALL)

Job PRIORITY* FS( User) Serv(XFctr) Res( Proc)
Weights -------- 1( 1) 1( 1) 1( 1)

46 113 6.9( -9.0) 1.2( 1.6) 91.9(120.0)
47 77 9.5( -9.0) 1.6( 1.5) 88.9( 84.0)

Percent Contribution -------- 8.0( 8.0) 1.4( 1.4) 90.6( 90.6)

FS refers to the fairshare factor. For both jobs, the user's usage is above the target usage, making the fairshare factor negative.

The exact weight given to each factor is subject to change as we attempt to find a balance between high utilization and fairness to all users.

The diagnose command will also tell you how much you and other users have been using the cluster over the past week.

$ diagnose -f
FairShare Information

Depth: 7 intervals Interval Length: 1:00:00:00 Decay Rate: 0.75

FS Policy: UTILIZEDPS
System FS Settings: Target Usage: 0.00 Flags: 0

FSInterval % Target 0 1 2 3 4 5 6
FSWeight ------- ------- 1.0000 0.7500 0.5625 0.4219 0.3164 0.2373 0.1780
TotalUsage 100.00 ------- 23779.0 4358.0 6114.8 10451.7 10853.7 6915.1 7861.7

USER
-------------
bvannemreddy 0.00 5.00 ------- ------- ------- ------- ------- ------- -------
ac.smiyawak* 16.74 5.00 18.66 32.48 0.75 16.23 0.71 17.28 27.11
bernard 0.09 5.00 ------- ------- ------- 0.80 0.00 0.02 0.00
ac.abarsan 2.17 5.00 2.08 ------- ------- ------- ------- 17.27 8.36

The asterisk indicates that the user is above their fairshare target, and that their jobs will be given a negative fairshare weight.

=== More qsub options ===

==== Receiving mail from Torque ====

You can have Torque automatically email you when a job begins, ends, or aborts.

qsub -m <mail-options>

The <mail-options> argument consists of either the single character 'n', or one or more of 'a', 'b', 'e'.

n No mail will be sent.
a Mail will be sent when the job aborts.
b Mail will be sent when the job begins.
e Mail will be sent when the job ends.

==== Redirecting job output ====

Torque will write the standard output of your jobscript to <jobname>.o<jobnumber>, and the standard error to <jobname>.e<jobnumber>. To write them both to the standard output file, submit the job with

qsub -joe ...

or in your jobscript use

#PBS -joe

To write standard output to a different file, submit the job with

qsub -o <some-file>

or in your jobscript:

#PBS -o <some-file>

These can be combined; so, for example, submitting a job with

qsub -joe -o logfile

will cause both standard output and standard error to be written to the file 'logfile' in the directory in which the job is running.

You can see the output file using the command qpeek:

qpeek <jobid>

qpeek has several useful options:

$ qpeek -?
qpeek: Peek into a job's output spool files

Usage: qpeek [options] JOBID

Options:
-c Show all of the output file ("cat", default)
-h Show only the beginning of the output file ("head")
-t Show only the end of the output file ("tail")
-f Show only the end of the file and keep listening ("tail -f")
-# Show only # lines of output
-e Show the stderr file of the job
-o Show the stdout file of the job (default)
-? Display this help message

==== Using fewer than 8 cores per node on Phoenix and fewer than 32 cores per node on Zephyr====

'''For Phoenix'''

* To request and use fewer than 8 cores per node, specify the number of cores you need with the '''ppn''' node property:

qsub -q quadcore -l nodes=<number of nodes>:ppn=<cores per node> ...

For example, to use 4 cores on each of 8 nodes, do

qsub -q quadcore -l nodes=8:ppn=4 ...

* By default, the scheduler will assign multiple jobs to a node as long as the node has resources available, and as long as the jobs are owned by the same user. This behavior can be prevented like this:

qsub -W x=NACCESSPOLICY:SINGLEJOB ...

so the previous command would be

qsub -q quadcore -l nodes=8:ppn=4 -W x=NACCESSPOLICY:SINGLEJOB ...

(It would probably make more sense to have SINGLEJOB as the default, but Maui for some reason does not allow that policy to be overridden.)

'''For Zephyr'''

* To request and use fewer than 32 cores per node, specify the number of cores you need with the '''ppn''' node property:

qsub -q batch -l nodes=<number of nodes>:ppn=<cores per node> ...

For example, to use 4 cores on each of 8 nodes, do

qsub -q batch -l nodes=8:ppn=4 ...

* By default, the scheduler will assign multiple jobs to a node as long as the node has resources available, and as long as the jobs are owned by the same user. This behavior can be prevented like this:

qsub -W x=NACCESSPOLICY:SINGLEJOB ...

so the previous command would be

qsub -q batch -l nodes=8:ppn=4 -W x=NACCESSPOLICY:SINGLEJOB ...

(It would probably make more sense to have SINGLEJOB as the default, but Maui for some reason does not allow that policy to be overridden.)

==== Forcing jobs to run sequentially ====

* To have a job start only after another job has finished, use the '''depend''' job attribute:

qsub -W depend=afterany:<some job-id> ...

For example,

qsub -W depend=afterany:1198 ...

Many other dependency relationships are possible; please see the [http://www.clusterresources.com/torquedocs21/commands/qsub.shtml qsub man page].

==== Passing variables to a jobscript ====

By default, no environment variables are directly exported from the submission environment to the execution environment in which a job runs. This means that you'll usually need to load any modules the job requires in your job script, or in your shell configuration files, as described above, or use one of the following techniques to pass variables to a job.

* To pass all environment variables:

export name=abc; export model=neon
qsub -V ...

* To pass only specified variables:

qsub -v name=abc,model=neon ...

or

export name=abc; export model=neon
qsub -v name,model ...

* You can also specify in a job script which variables Torque should import into the execution environment from the submission environment. To import all environment variables:

#PBS -V

or to import only specified variables:

#PBS -v name,model

Note that you cannot assign values to the variables this way.

=== Job and cluster info ===

* To see a list of jobs which are running, queued, and blocked, use the Maui command showq:

showq [-r | -i | -b ]

For example,

$ showq -r
JobName S Par Effic XFactor Q User Group MHost Procs Remaining StartTime

67174 R DEF 100.00 0.0 pr joe joe n101-ib 1 00:23:32 Wed Apr 28 14:05:12
67233+ R DEF 182.60 1.1 pr jan jan n082-ib 8 14:53:30 Thu Apr 29 13:37:49
66942- R DEF 49.71 0.3 pr lin lin n056-ib 64 18:59:14 Tue Apr 27 08:41:35

The "+" indicates that the job was [[#How_the_scheduler_determines_which_job_to_run_next | backfilled]]; the "—" that the job is [[Using_the_cluster#Quality_of_Service_.28QOS.29 | preemptible]].

showq lists queued (Idle) jobs in descending [[#How_the_scheduler_determines_which_job_to_run_next | priority]] order. showq may show a job as running for up to a minute before it actually starts. To see if it's actually running in that first minute, use Torque's qstat:

qstat -a

* To see how many cores are currently free for a job of a given walltime in each queue, use qbf:

qbf -d <walltime>

* To get more detailed statistics about current cluster usage, use our command qsum:

qsum

* Use pbstop for a graphical display of cluster usage:

pbstop [-?]

* pestat will give node-by-node cluster information:

pestat [-?]

* To get a lot of info from Torque about a running or queued job:

qstat -f <job-id>

* To diagnose problems with a job, you'll need to use both the Torque command tracejob and the Maui command checkjob, and possibly the Maui command diagnose:

tracejob <job-id>
checkjob [-v] <job-id>
diagnose -j <job-id>

* If you want to find out when Maui thinks your job will run, use showstart:

showstart <job-id>

$ showstart 56
job 56 requires 32 procs for 1:40:00
Earliest start in 1:32:48 on Wed Nov 25 18:02:37
Earliest completion in 3:12:48 on Wed Nov 25 19:42:37

* If you want to find out how small and how short you need to make a job for it to run immediately in a particular queue, use the Maui command showbf:

showbf [-f <node-feature>] [-n nodecount] [-d walltime]

where <node-feature> is one of quadcore, eightgb, or thirtytwogb. ("bf" is for backfill, the algorithm used to schedule short and small jobs without interfering with higher priority jobs.) Maui unfortunately doesn't take into account all of our scheduling policies, so showbf's core counts may be too high; nor does it know about our queue walltime limits.

* We have our own utility, qbf, which summarizes the output of showbf for each queue:

$ qbf
Queue Cores Time
quadcore 112 5:49:09
quadcore 40 3:00:00:00
thirtytwogb 8 1:00:00:00

This means that a 40 core job of any duration will run immediately, but a job requesting more than 40 cores will run immediately only if it requests less than 5 hours, 49 minutes. qbf is, however, subject to the same caveat as showbf.

* To get a list of the nodes on which jobs are running (use -s to sort the list):

jobnodes [-s] <jobid> ...

$ jobnodes -s 45124 45873
n008,n015

* To get a list of the nodes on which your jobs are running (use -s to sort the list):

mynodes [-s]

$ mynodes -s
n010,n097

* To see if your application is still running on your nodes:

$ pdsh -w $(mynodes) ps

n046: PID TTY TIME CMD
n046: 13056 ? 00:00:00 bash
n046: 13717 ? 00:00:00 bash
n046: 13721 ? 00:00:00 sshd
n046: 13722 ? 00:00:00 ps

For all of these commands, only the numerical portion of the job-id needs to be specified.

==== qsub-stat ====

qsub-stat is available only on Phoenix and requires the ls-dyna/base module to be loaded.

Most of the commands for tracking your jobs and checking cluster status can be accessed via graphical application qsub-stat. Please refer to section on NoMachine to learn how to run graphical sessions on the TRACC Cluster. qsub-stat has three tabs. The first one shows percentage usage of the two queues (8GB and 32GB) and the disk space usage. Once you move mouse cursor on the meter you will get more detailed information on usage of the queues and the disk space.

[[Image:600px-Resources2.png|thumb|none|left|600px|alt=Resources tab|Resources tab]]

In the Terminal tab majority of explained previously statistics commands can be called through mouse menu. Press and hold right or left mouse button to have access to the commands' list.

[[Image:600px-Terminal2.png|thumb|none|left|600px|alt=Terminal tab|Terminal tab]]

The available commands are:

-------------------------------------------
- Cluster Usage (showq):
- showq
- showq - r (running)
- showq - b (blocked)
- Cluster Usage (qsum)
- Cluster Usage (qstat)
- Available Resources (qbf)
-------------------------------------------
- LS-DYNA Usage (qdyna)
- Madymo Usage (license server report)
- Fluent Usage (license server report)
-------------------------------------------
- Current Login Node activity (htop)
- Node activity (pbstop)
- Node activity (pestat)
-------------------------------------------
- See My Jobs
-------------------------------------------

Current usage of the cluster can also be viewed in the Job Status tab. When you press and hold right or left mouse button on a job, you will see a menu of commands that you can apply to it (Some of them will work only on your own jobs). The available commands are:

-------------------------------------------
- Show additional info:
- qstat -f
- chcekjob
- tracejob
- diagnosejob -j
- showstart
-------------------------------------------
- open standard output
- statistics on master node (htop)
- ssh to master node
-------------------------------------------
- open work directory in browser
- open work directory in terminal
-------------------------------------------
- delete job (qdel)
-------------------------------------------

[[Image:600px-Job_status2.png|thumb|none|left|600px|alt=Job Status tab|Job Status tab]]

Htop command shows live statistics on the master node of the current job.
[[Image:600px-Htop.png|thumb|none|left|600px|alt=Htop|Htop]]

=== Deleting a job ===

To delete a job, use Torque's qdel or Maui's canceljob, which simply calls qdel itself:

qdel <job-id>
canceljob <job-id>

Torque will delete the job immmediately with either command, but Maui may not know about the deletion for up to a minute, during which time it will continue to show up in the output of showq.

If either Torque or Maui tells you it can't delete the job, please send us an email and we'll delete it for you.

Only the numerical portion of the job-id needs to be specified.

=== Interactive Jobs ===

You can get interactive access to a compute node with Torque by using the -I switch to qsub:

$ qsub -I ...
qsub: waiting for job nnnn.host1 to start
qsub: job nnnn.host1 ready
$ cd $PBS_O_WORKDIR

If you want to run a GUI application on the compute node, use the -X switch:

qsub -I -X ...

For this to work, you will need to either use the [[#Graphical_Applications_using_NoMachine_NX | NX client]] to connect to the cluster, or enable trusted [[#Graphical_Applications_using_X | X11]] forwarding when you ssh to the cluster. Windows SSH clients will allow you to enable forwarding in the preferences for the connection. Under Linux and MacOS, you would do this:

$ ssh -Y login.tracc.anl.gov

Also see [[#Accessing_a_compute_node_directly | below]] for an alternative means of directly accessing computes nodes.

=== Using GUI applications on a compute node ===

If you want to use a GUI application on a compute node, you will need to enable trusted [[#Graphical_Applications_using_X | X11]] forwarding when you ssh to the cluster, and again when you ssh to the compute node (as described above). Windows SSH clients will allow you to enable forwarding in the preferences for the connection. Under Linux and MacOS, you would do this:

$ ssh -Y login.tracc.anl.gov

After logging in to a login node, you would then do this to access the compute node:

$ ssh -Y n001

If you're using the [[#Graphical_Applications_using_NoMachine_NX | NX client]] to connect to the cluster, you won't need to ssh to a login node, but you will still need to ssh to a compute node.

=== Accessing a compute node directly ===

The easiest way to do this is by having Torque give you [[#Interactive_jobs | interactive]] access to a node. But if you want to start a process running on a compute node and have it continue to run after you log out, use the method described here.

* You can access any compute node which is running one of your jobs with the command

ssh <compute-node>

You can use the command

qstat -n1 <job-id>

to find out which nodes are running <job-id>.

* If you want Torque to assign you nodes, but don't need it to run a job on your behalf, you can do (for example)

echo sleep 24h | qsub -l select=1,walltime=24:00:00

and then ssh to the node.

=== Using licensed software ===

For some of our software we have a limited number of licenses. To ensure that there will be sufficient licenses available when Maui schedules your job, you will need to submit your job under an application-specific account. Any application wrapper scripts we have will take care of this for you, but if you need to specify the account manually, you would do it like this:

* LS-DYNA:

qsub -A dyna ...

=== Debugging a job ===

Some things to keep in mind or to try when your job doesn't run as expected:

* Jobs don't inherit any environment variables from the submission environment. This means that if your job requires that PATH, LD_LIBRARY_PATH, MPI_HOME, or any other variable be set, you'll need to set it either in the job script, either directly or with a module, or in your shell configuration files.

* Try running your job interactively on a compute node, like this:

$ qsub -I ...
qsub: waiting for job nnnn.host1 to start
qsub: job nnnn.host1 ready
$ cd $PBS_O_WORKDIR

Now run your job script or start up a debugger with your program, just as you would on a login node.

* Try some simple jobs, and look at the standard output and standard error files (STDIN.o<job-id> and STDIN.e<job-id>):

echo | qsub
echo env | qsub

* If you want to see the output of your job as it's being produced, use qpeek:

qpeek <jobid>

=== Undeliverable job output ===

If Torque cannot copy the files containing the output of your job to the job's working directory, the files will eventually be copied to /mnt/gpfs/undelivered. They will be deleted after 10 days.

=== Requeuing of jobs ===

If your job dies due to a problem with the cluster, it will not automatically be requeued. If you would like it to be, do

qsub -r y ...

=== Reserving the cluster ===

Users cannot directly reserve the cluster, but if you think you might have a need for a reservation, please contact us.
[mailto:TRACC_Director@anl.gov Click to send email] or via telephone: 630-252-8224 as far in advance as possible.

:::::::::::::::[[New main|'''Return To Main Page''']]

Compiling Your Code

2025-03-09T01:28:13Z

Amiot:

if you need help to compile your code, please contact us by [mailto:TRACC_Director@anl.gov clicking to send email].

:::::::::::::::[[New main|'''Return To Main Page''']]

Backups

2025-03-09T01:27:31Z

Amiot:

Files & Storage

2025-03-09T01:26:46Z

Amiot:

Login Nodes Versus Compute Nodes

2025-03-09T01:25:33Z

Amiot:

Setting Up Your Environment

2025-03-09T01:24:50Z

Amiot:

=== Shells ===

* Your account will be set up with the bash shell, unless you asked for a different shell when you requested your account. You can find out your current shell with the command <code>echo $SHELL</code>.

* If you'd like a different shell, please contact us. [mailto:TRACC_Director@anl.gov Click to send email].

* The commands <code>chsh</code> and <code>ypchsh</code> don't work on the cluster, and have been disabled.

=== Accessing application software with modules ===

* The cluster uses the [http://modules.sourceforge.net Environment Modules] package to manage access to application software.

* modules documentation is available with the command <code>man module</code>.

* To set up your environment so that an application can be used, use the command

module load <module>

at the command line or in your shell's configuration files.

* You can undo the effect of a module load with

module unload <module>

* It's generally not a good idea to load modules for more than one version of the same application. Instead, swap the modules:

module swap some-application/stable some-application/beta

* Your .bashrc or .cshrc contains sample commands for loading the most common modules. Please see the instructions in those files, or in /etc/skel/.bashrc or /etc/skel/.cshrc, which will be the latest versions of the default configuration files.

* Most modules are one of these two forms:

<application type>/<application>/<version>
<application>/<version>

for example

compiler/pathscale/3.1-147
starccm/15.06.008-R8

Omitting the version number will load the latest stable version of the module.

To see what modules are currently loaded in your environment, do

module list

To see what modules are available, do

module avail

* Modules with "-alpha" or "-beta" appended to the name may not work, either because the software or module is broken.

* To use modules in shell scripts, including Torque jobs scripts, see [[Running Your Code#Job_scripts| below]].

=== Changing ARROW Computing Queues ===
There are currently several computing queues that are available to schedule and some of these queues can only be scheduled by one scheduler (Torque) and the others by a second scheduler (PBS). The nodes scheduled by the queues vary from queue to queue. Be aware that node characteristics (RAM, number of processors, etc) vary from queue to queue. The Torque scheduler is availale by default; the PBS scheduler is only available by loading its environmental variable:

The queues "PBS Computing Queues" are available after the environmental specifying the PBS Scheduler is initiated

<pre>
module load pbs/pbs
</pre>
You can review the queues that are available and which scheduler to use to [[ARROW Cluster under construction |'''schedule them by clicking here.''']]

=== File permissions ===

By default, you will be in your own Unix group, and your files will be readable and writable only by you (that is, the default umask is 027). If you want files to be readable by everyone working on your project, you will need to

* Have your project leader request that we create a Unix group for the project. They also need to let us know who should be in the group.

* Change the group ownership of the files you want to be group-readable:

chgrp <project-group> <filename>

* Make the file group-readable, and, if user-executable, group-executable:

chmod g+rX <filename>

If you want all your new files to be readable by everyone in your group, you'll need to

* Ask us to change your default group. To contact us
[mailto:TRACC_Director@anl.gov click to send email] or via telephone: 630-252-8224.

Note that a change in your primary group will not affect pre-existing files. To change the group ownership of all your files, do this:

chgrp -R <new-group> ~
chmod -R g+rX ~
chgrp -R $USER ~/.ssh
chmod -R g-rx ~/.ssh

The last two commands are necessary for SSH security; you will lock yourself out of the cluster if you omit them.

:::::::::::::::[[New main|'''Return To Main Page''']]

Transferring Files

2025-03-09T01:23:20Z

Amiot:

Transferring Files

2025-03-09T01:21:59Z

Amiot:

Transferring Files

2025-03-09T01:20:38Z

Amiot:

Becoming A User

2025-03-09T01:18:44Z

Amiot:

Setting Up Your Environment

2025-03-08T23:14:57Z

Amiot: /* Changing ARROW Computing Queues */

=== Shells ===

* Your account will be set up with the bash shell, unless you asked for a different shell when you requested your account. You can find out your current shell with the command <code>echo $SHELL</code>.

* If you'd like a different shell, please contact us. [mailto:TRACC_Director@anl.gov Click to send email].

* The commands <code>chsh</code> and <code>ypchsh</code> don't work on the cluster, and have been disabled.

=== Accessing application software with modules ===

* The cluster uses the [http://modules.sourceforge.net Environment Modules] package to manage access to application software.

* modules documentation is available with the command <code>man module</code>.

* To set up your environment so that an application can be used, use the command

module load <module>

at the command line or in your shell's configuration files.

* You can undo the effect of a module load with

module unload <module>

* It's generally not a good idea to load modules for more than one version of the same application. Instead, swap the modules:

module swap some-application/stable some-application/beta

* Your .bashrc or .cshrc contains sample commands for loading the most common modules. Please see the instructions in those files, or in /etc/skel/.bashrc or /etc/skel/.cshrc, which will be the latest versions of the default configuration files.

* Most modules are one of these two forms:

<application type>/<application>/<version>
<application>/<version>

for example

compiler/pathscale/3.1-147
starccm/15.06.008-R8

Omitting the version number will load the latest stable version of the module.

To see what modules are currently loaded in your environment, do

module list

To see what modules are available, do

module avail

* Modules with "-alpha" or "-beta" appended to the name may not work, either because the software or module is broken.

* To use modules in shell scripts, including Torque jobs scripts, see [[Running Your Code#Job_scripts| below]].

=== Changing ARROW Computing Queues ===
There are currently several computing queues that are available to schedule and some of these queues can only be scheduled by one scheduler (Torque) and the others by a second scheduler (PBS). The nodes scheduled by the queues vary from queue to queue. Be aware that node characteristics (RAM, number of processors, etc) vary from queue to queue. The Torque scheduler is availale by default; the PBS scheduler is only available by loading its environmental variable:

The queues "PBS Computing Queues" are available after the environmental specifying the PBS Scheduler is initiated

<pre>
module load pbs/pbs
</pre>
You can review the queues that are available and which scheduler to use to [[ARROW Cluster under construction |'''schedule them by clicking here.''']]

=== File permissions ===

By default, you will be in your own Unix group, and your files will be readable and writable only by you (that is, the default umask is 027). If you want files to be readable by everyone working on your project, you will need to

* Have your project leader request that we create a Unix group for the project. They also need to let us know who should be in the group.

* Change the group ownership of the files you want to be group-readable:

chgrp <project-group> <filename>

* Make the file group-readable, and, if user-executable, group-executable:

chmod g+rX <filename>

If you want all your new files to be readable by everyone in your group, you'll need to

* Ask us to change your default group. To contact us
[mailto:TRACC_Director@anl.gov click to send email] or via telephone: 630-252-8224.

Note that a change in your primary group will not affect pre-existing files. To change the group ownership of all your files, do this:

chgrp -R <new-group> ~
chmod -R g+rX ~
chgrp -R $USER ~/.ssh
chmod -R g-rx ~/.ssh

The last two commands are necessary for SSH security; you will lock yourself out of the cluster if you omit them.

Setting Up Your Environment

2025-03-08T23:05:17Z

Amiot: /* Changing ARROW Computing Queues */

=== Shells ===

* Your account will be set up with the bash shell, unless you asked for a different shell when you requested your account. You can find out your current shell with the command <code>echo $SHELL</code>.

* If you'd like a different shell, please contact us. [mailto:TRACC_Director@anl.gov Click to send email].

* The commands <code>chsh</code> and <code>ypchsh</code> don't work on the cluster, and have been disabled.

=== Accessing application software with modules ===

* The cluster uses the [http://modules.sourceforge.net Environment Modules] package to manage access to application software.

* modules documentation is available with the command <code>man module</code>.

* To set up your environment so that an application can be used, use the command

module load <module>

at the command line or in your shell's configuration files.

* You can undo the effect of a module load with

module unload <module>

* It's generally not a good idea to load modules for more than one version of the same application. Instead, swap the modules:

module swap some-application/stable some-application/beta

* Your .bashrc or .cshrc contains sample commands for loading the most common modules. Please see the instructions in those files, or in /etc/skel/.bashrc or /etc/skel/.cshrc, which will be the latest versions of the default configuration files.

* Most modules are one of these two forms:

<application type>/<application>/<version>
<application>/<version>

for example

compiler/pathscale/3.1-147
starccm/15.06.008-R8

Omitting the version number will load the latest stable version of the module.

To see what modules are currently loaded in your environment, do

module list

To see what modules are available, do

module avail

* Modules with "-alpha" or "-beta" appended to the name may not work, either because the software or module is broken.

* To use modules in shell scripts, including Torque jobs scripts, see [[Running Your Code#Job_scripts| below]].

=== Changing ARROW Computing Queues ===
There are currently several computing queues that are available to schedule and some of these queues can only be scheduled by one scheduler (Torque) and the others by a second scheduler (PBS). The nodes scheduled by the queues vary from queue to queue. Be aware that node characteristics (RAM, number of processors, etc) vary from queue to queue. The Torque scheduler is availale by default; the PBS scheduler is only available by loading its environmental variable:

The queues "PBS Computing Queues" are available after the environmental specifying the PBS Scheduler is initiated

<pre>
module load pbs/pbs
</pre>
You can review the queues that are available and which scheduler to use to [[ARROW Cluster |'''schedule them by clicking here.''']]

=== File permissions ===

By default, you will be in your own Unix group, and your files will be readable and writable only by you (that is, the default umask is 027). If you want files to be readable by everyone working on your project, you will need to

* Have your project leader request that we create a Unix group for the project. They also need to let us know who should be in the group.

* Change the group ownership of the files you want to be group-readable:

chgrp <project-group> <filename>

* Make the file group-readable, and, if user-executable, group-executable:

chmod g+rX <filename>

If you want all your new files to be readable by everyone in your group, you'll need to

* Ask us to change your default group. To contact us
[mailto:TRACC_Director@anl.gov click to send email] or via telephone: 630-252-8224.

Note that a change in your primary group will not affect pre-existing files. To change the group ownership of all your files, do this:

chgrp -R <new-group> ~
chmod -R g+rX ~
chgrp -R $USER ~/.ssh
chmod -R g-rx ~/.ssh

The last two commands are necessary for SSH security; you will lock yourself out of the cluster if you omit them.

Setting Up Your Environment

2025-03-08T22:40:13Z

Amiot: /* Changing ARROW Computing Queues */

=== Shells ===

* Your account will be set up with the bash shell, unless you asked for a different shell when you requested your account. You can find out your current shell with the command <code>echo $SHELL</code>.

* If you'd like a different shell, please contact us. [mailto:TRACC_Director@anl.gov Click to send email].

* The commands <code>chsh</code> and <code>ypchsh</code> don't work on the cluster, and have been disabled.

=== Accessing application software with modules ===

* The cluster uses the [http://modules.sourceforge.net Environment Modules] package to manage access to application software.

* modules documentation is available with the command <code>man module</code>.

* To set up your environment so that an application can be used, use the command

module load <module>

at the command line or in your shell's configuration files.

* You can undo the effect of a module load with

module unload <module>

* It's generally not a good idea to load modules for more than one version of the same application. Instead, swap the modules:

module swap some-application/stable some-application/beta

* Your .bashrc or .cshrc contains sample commands for loading the most common modules. Please see the instructions in those files, or in /etc/skel/.bashrc or /etc/skel/.cshrc, which will be the latest versions of the default configuration files.

* Most modules are one of these two forms:

<application type>/<application>/<version>
<application>/<version>

for example

compiler/pathscale/3.1-147
starccm/15.06.008-R8

Omitting the version number will load the latest stable version of the module.

To see what modules are currently loaded in your environment, do

module list

To see what modules are available, do

module avail

* Modules with "-alpha" or "-beta" appended to the name may not work, either because the software or module is broken.

* To use modules in shell scripts, including Torque jobs scripts, see [[Running Your Code#Job_scripts| below]].

=== Changing ARROW Computing Queues ===
There are currently several computing queues that are available to schedule. Be aware that all nodes in some queues have the same characteristics (RAM, etc) while some queues have nodes with different characteristics. Thus jobs using those queues must specify the node names that are to be used, further some queues can only scheduled by one scheduler (Torque) and the other by a different scheduler (PBS). 
The queues "Torque Computing Queues" are scheduled by the Torque scheduler and is available by default. 

The queues "PBS Computing Queues" are available after the environmental specifying the PBS Scheduler is initiated

<pre>
module load pbs/pbs
</pre>
You can review the queues that are available and which scheduler to use to [[ARROW Cluster |'''schedule them by clicking here.''']]

=== File permissions ===

By default, you will be in your own Unix group, and your files will be readable and writable only by you (that is, the default umask is 027). If you want files to be readable by everyone working on your project, you will need to

* Have your project leader request that we create a Unix group for the project. They also need to let us know who should be in the group.

* Change the group ownership of the files you want to be group-readable:

chgrp <project-group> <filename>

* Make the file group-readable, and, if user-executable, group-executable:

chmod g+rX <filename>

If you want all your new files to be readable by everyone in your group, you'll need to

* Ask us to change your default group. To contact us
[mailto:TRACC_Director@anl.gov click to send email] or via telephone: 630-252-8224.

Note that a change in your primary group will not affect pre-existing files. To change the group ownership of all your files, do this:

chgrp -R <new-group> ~
chmod -R g+rX ~
chgrp -R $USER ~/.ssh
chmod -R g-rx ~/.ssh

The last two commands are necessary for SSH security; you will lock yourself out of the cluster if you omit them.

Setting Up Your Environment

2025-03-08T22:38:19Z

Amiot: /* Changing ARROW Computing Queues */

=== Shells ===

* Your account will be set up with the bash shell, unless you asked for a different shell when you requested your account. You can find out your current shell with the command <code>echo $SHELL</code>.

* If you'd like a different shell, please contact us. [mailto:TRACC_Director@anl.gov Click to send email].

* The commands <code>chsh</code> and <code>ypchsh</code> don't work on the cluster, and have been disabled.

=== Accessing application software with modules ===

* The cluster uses the [http://modules.sourceforge.net Environment Modules] package to manage access to application software.

* modules documentation is available with the command <code>man module</code>.

* To set up your environment so that an application can be used, use the command

module load <module>

at the command line or in your shell's configuration files.

* You can undo the effect of a module load with

module unload <module>

* It's generally not a good idea to load modules for more than one version of the same application. Instead, swap the modules:

module swap some-application/stable some-application/beta

* Your .bashrc or .cshrc contains sample commands for loading the most common modules. Please see the instructions in those files, or in /etc/skel/.bashrc or /etc/skel/.cshrc, which will be the latest versions of the default configuration files.

* Most modules are one of these two forms:

<application type>/<application>/<version>
<application>/<version>

for example

compiler/pathscale/3.1-147
starccm/15.06.008-R8

Omitting the version number will load the latest stable version of the module.

To see what modules are currently loaded in your environment, do

module list

To see what modules are available, do

module avail

* Modules with "-alpha" or "-beta" appended to the name may not work, either because the software or module is broken.

* To use modules in shell scripts, including Torque jobs scripts, see [[Running Your Code#Job_scripts| below]].

=== Changing ARROW Computing Queues ===
There are currently several computing queues that are available to schedule. Be aware that all nodes in some queues have the same characteristics (RAM, etc) while some queues have nodes with different characteristics. Thus jobs using those queues must specify the node names that are to be used, further some queues can only scheduled by one scheduler (Torque) and the other by a different scheduler (PBS). 
The queues "Torque Computing Queues" are scheduled by the Torque scheduler and is available by default. 

The queues "PBS Computing Queues" are available after the environmental specifying the PBS Scheduler is initiated

<pre>
module load pbs/pbs
</pre>
You can review the queues that are available and which scheduler to use to [[ARROW Cluster |'''schedule them by clicking here.''']]

One | '''Introduction to the ARROW Cluster''']] 
:* [[ARROW Cluster#ARROW Queues | '''ARROW Computing Queues''']] 

=== File permissions ===

By default, you will be in your own Unix group, and your files will be readable and writable only by you (that is, the default umask is 027). If you want files to be readable by everyone working on your project, you will need to

* Have your project leader request that we create a Unix group for the project. They also need to let us know who should be in the group.

* Change the group ownership of the files you want to be group-readable:

chgrp <project-group> <filename>

* Make the file group-readable, and, if user-executable, group-executable:

chmod g+rX <filename>

If you want all your new files to be readable by everyone in your group, you'll need to

* Ask us to change your default group. To contact us
[mailto:TRACC_Director@anl.gov click to send email] or via telephone: 630-252-8224.

Note that a change in your primary group will not affect pre-existing files. To change the group ownership of all your files, do this:

chgrp -R <new-group> ~
chmod -R g+rX ~
chgrp -R $USER ~/.ssh
chmod -R g-rx ~/.ssh

The last two commands are necessary for SSH security; you will lock yourself out of the cluster if you omit them.

Setting Up Your Environment

2025-03-08T22:32:52Z

Amiot:

=== Shells ===

* Your account will be set up with the bash shell, unless you asked for a different shell when you requested your account. You can find out your current shell with the command <code>echo $SHELL</code>.

* If you'd like a different shell, please contact us. [mailto:TRACC_Director@anl.gov Click to send email].

* The commands <code>chsh</code> and <code>ypchsh</code> don't work on the cluster, and have been disabled.

=== Accessing application software with modules ===

* The cluster uses the [http://modules.sourceforge.net Environment Modules] package to manage access to application software.

* modules documentation is available with the command <code>man module</code>.

* To set up your environment so that an application can be used, use the command

module load <module>

at the command line or in your shell's configuration files.

* You can undo the effect of a module load with

module unload <module>

* It's generally not a good idea to load modules for more than one version of the same application. Instead, swap the modules:

module swap some-application/stable some-application/beta

* Your .bashrc or .cshrc contains sample commands for loading the most common modules. Please see the instructions in those files, or in /etc/skel/.bashrc or /etc/skel/.cshrc, which will be the latest versions of the default configuration files.

* Most modules are one of these two forms:

<application type>/<application>/<version>
<application>/<version>

for example

compiler/pathscale/3.1-147
starccm/15.06.008-R8

Omitting the version number will load the latest stable version of the module.

To see what modules are currently loaded in your environment, do

module list

To see what modules are available, do

module avail

* Modules with "-alpha" or "-beta" appended to the name may not work, either because the software or module is broken.

* To use modules in shell scripts, including Torque jobs scripts, see [[Running Your Code#Job_scripts| below]].

=== Changing ARROW Computing Queues ===
There are currently several computing queues that are available to schedule. Be aware that all nodes in some queues have the same characteristics (RAM, etc) while some queues have nodes with different characteristics. Thus jobs using those queues must specify the node names that are to be used, further some queues can only scheduled by one scheduler (Torque) and the other by a different scheduler (PBS). 
The queues "Torque Computing Queues" are scheduled by the Torque scheduler and is available by default. 

The queues "PBS Computing Queues" are available after the environmental specifying the PBS Scheduler is initiated

<pre>
module load pbs/pbs
</pre>

=== File permissions ===

By default, you will be in your own Unix group, and your files will be readable and writable only by you (that is, the default umask is 027). If you want files to be readable by everyone working on your project, you will need to

* Have your project leader request that we create a Unix group for the project. They also need to let us know who should be in the group.

* Change the group ownership of the files you want to be group-readable:

chgrp <project-group> <filename>

* Make the file group-readable, and, if user-executable, group-executable:

chmod g+rX <filename>

If you want all your new files to be readable by everyone in your group, you'll need to

* Ask us to change your default group. To contact us
[mailto:TRACC_Director@anl.gov click to send email] or via telephone: 630-252-8224.

Note that a change in your primary group will not affect pre-existing files. To change the group ownership of all your files, do this:

chgrp -R <new-group> ~
chmod -R g+rX ~
chgrp -R $USER ~/.ssh
chmod -R g-rx ~/.ssh

The last two commands are necessary for SSH security; you will lock yourself out of the cluster if you omit them.

Setting Up Your Environment

2025-02-23T22:05:11Z

Amiot: /* Shells */

=== Shells ===

* Your account will be set up with the bash shell, unless you asked for a different shell when you requested your account. You can find out your current shell with the command <code>echo $SHELL</code>.

* If you'd like a different shell, please contact us. [mailto:TRACC_Director@anl.gov Click to send email].

* The commands <code>chsh</code> and <code>ypchsh</code> don't work on the cluster, and have been disabled.

=== Accessing application software with modules ===

* The cluster uses the [http://modules.sourceforge.net Environment Modules] package to manage access to application software.

* modules documentation is available with the command <code>man module</code>.

* To set up your environment so that an application can be used, use the command

module load <module>

at the command line or in your shell's configuration files.

* You can undo the effect of a module load with

module unload <module>

* It's generally not a good idea to load modules for more than one version of the same application. Instead, swap the modules:

module swap some-application/stable some-application/beta

* Your .bashrc or .cshrc contains sample commands for loading the most common modules. Please see the instructions in those files, or in /etc/skel/.bashrc or /etc/skel/.cshrc, which will be the latest versions of the default configuration files.

* Most modules are one of these two forms:

<application type>/<application>/<version>
<application>/<version>

for example

compiler/pathscale/3.1-147
starccm/15.06.008-R8

Omitting the version number will load the latest stable version of the module.

To see what modules are currently loaded in your environment, do

module list

To see what modules are available, do

module avail

* Modules with "-alpha" or "-beta" appended to the name may not work, either because the software or module is broken.

* To use modules in shell scripts, including Torque jobs scripts, see [[Running Your Code#Job_scripts| below]].

=== File permissions ===

By default, you will be in your own Unix group, and your files will be readable and writable only by you (that is, the default umask is 027). If you want files to be readable by everyone working on your project, you will need to

* Have your project leader request that we create a Unix group for the project. They also need to let us know who should be in the group.

* Change the group ownership of the files you want to be group-readable:

chgrp <project-group> <filename>

* Make the file group-readable, and, if user-executable, group-executable:

chmod g+rX <filename>

If you want all your new files to be readable by everyone in your group, you'll need to

* Ask us to change your default group. To contact us
[mailto:TRACC_Director@anl.gov click to send email] or via telephone: 630-252-8224.

Note that a change in your primary group will not affect pre-existing files. To change the group ownership of all your files, do this:

chgrp -R <new-group> ~
chmod -R g+rX ~
chgrp -R $USER ~/.ssh
chmod -R g-rx ~/.ssh

The last two commands are necessary for SSH security; you will lock yourself out of the cluster if you omit them.

How to Connect

2025-01-14T21:28:05Z

Amiot: /* SSH */

You need an SSH client to connect to Arrow. No other connection protocols are supported. There are 4 general purpose login nodes that you can connect to:

*login1.arrow.tracc.anl.gov
*login2.arrow.tracc.anl.gov
*login3.arrow.tracc.anl.gov
*login4.arrow.tracc.anl.gov

When connecting, please pick a random entry from the above list to evenly distribute users over multiple machines.

For users that work at Argonne or that have an actual Argonne ANL Domain account, you use your assigned account name and password to connect to these login nodes. Just use the ANL Domain user name, without the ANL prefix.

For external collaborators, the bulk of our users, you want to use your external collaborator account that was assigned to you when you applied for an account. These accounts all start with the letters "ac" followed by a period, and then followed by a specific account name. The entire account name, including the leading "ac." portion must be used to connect.

Passwords expire on a regular basis (every 6 months or so), and you will need to update your password with Argonne Account Services. You can call 630-252-9999 if you need any help or detailed instructions.

You will not be able to connect until we have your public IP address on file in our system. Your account will be linked to a few machines with known IP addresses, and if you need to connect from another machine, please contact us to add that machine to your account. Details can be found below.

== How to Connect: Details ==

=== SSH ===

* Access to the TRACC cluster requires use of an SSH client. Free SSH clients are available for Windows, are included or available for MacOS, and all varieties of Linux or UNIX. SSH clients can be terminal applications to work with command lines (e.g. Putty on Windows), or they can be file transfer utilities (WinSCP, FileZilla). In any case, cluster connections are all using the SSH protocol (including sftp).

* If you're using a simple graphical application on the cluster through an SSH connection and you are either connecting from a Mac or Linux desktop machine or you have an X server installed on your Windows machine, you may want to enable X11 forwarding to your client. This is a method that uses strictly the old X protocol, and performance is very limited (due to the SSH encryption overhead). You should consider using one of the remote GUI options listed below instead if you can. If you want to keep it simple though, you can do this (using typically a corresponding check box in Windows clients):

ssh -X username@login1.arrow.tracc.anl.gov'''
'''--[[User:Amiot|Amiot]] ([[User talk:Amiot|talk]]) 15:28, January 14, 2025 (CST)
* Alternatively, client software like the [[Graphical Applications#GRAPHICAL APPLICATIONS Using NoMachine NX | NoMachine NX client]] can be used to connect, providing a full graphical user interface. This requires the installation of the free NoMachine client software.

* Another alternative is the [[Graphical Applications#GRAPHICAL APPLICATIONS Using NoMachine NX | free X2Go client]], which provides a near identical user experience. Details can be found [[Graphical Applications#GRAPHICAL APPLICATIONS Using NoMachine NX | here]].

=== Access restricted to authorized hosts ===

* Access to the cluster is currently restricted to pre-approved hosts. Please contact us [mailto:TRACC_Director@anl.gov clicking to send email] or via telephone at 630-252-8224 with the hostnames of the systems from which you would like to access the cluster. Argonne users with Argonne Domain accounts can connect through Argonne's VPN service; please contact us by [mailto:TRACC_Director@anl.gov clicking to send email] or via telephone: 630-252-8224 for details.

* Under Windows, you can find out your host name with the command "ipconfig". Note that IP addresses beginning with "10." or "192.168." are private addresses assigned by your home router, and those are not accessible from outside your home network. If you have such an address, you can usually find your public IP address at (for example) http://whatismyip.com.

TRACC Cluster Software

2024-12-24T21:10:30Z

Amiot: /* Under Investigation */

TRACC Cluster Software

2024-12-24T21:09:17Z

Amiot: /* Under Investigation */

Python

2021-07-30T19:08:44Z

Amiot:

--[[User:Amiot|Amiot]] ([[User talk:Amiot|talk]]) 16:20, July 6, 2021 (CDT)== Arrow ==
The default version of Python on Arrow is 2.7.5. and does not require a module to be loaded. You can start the interactive version by entering the command
python2

To run the batch version, start your script with
#!/usr/bin/env python2

The default version of python3 (3.6.8) along with sympy, numpy, and ipython are also installed, but you must load the python libraries. You can use the interactive version by entering the commands
module load python/3.8.2
python3

You can use the batch version by starting your script with
module load python/3.8.2
#!/usr/bin/env python2

You can see general information about modules [[Setting Up Your Environment#Accessing application software with modules| here.]]