HPC/Submitting and Managing Jobs: Difference between revisions

From CNM Wiki
< HPC
Jump to navigation Jump to search
 
(136 intermediate revisions by the same user not shown)
Line 1: Line 1:
== Directories and Environment ==
[[Image:HPC job flow.png|right|400px|thumb|Processing flow of a typical job.]]
__TOC__
== Introduction ==
To accept and run jobs, Carbon uses the [http://en.wikipedia.org/wiki/TORQUE_Resource_Manager Torque Resource Manager],
which is a variant of [http://en.wikipedia.org/wiki/Portable_Batch_System PBS].
A separate software component called a [http://en.wikipedia.org/wiki/Job_scheduler ''scheduler''] chooses the nodes and the start time for jobs according to the job parameters, resource availability, and configuration. Carbon uses the [http://www.adaptivecomputing.com/resources/docs/mwm/6-1-9/help.htm Moab Workload Manager] as scheduler. The resources used by each job (primarily the CPU time) are tracked and managed by the
[http://www.adaptivecomputing.com/resources/docs/gold/index.php Gold Allocation Manager].


First read: '''[[HPC/Directories|directory configuration]]'''.
=== Job processing ===
A job consists of a [[#Job Script|job script]] and applications called therein. The figure at the top shows schematically how a typical job is processed.
Unless otherwise told, Torque will execute the job script on your behalf on the first node allocated by Moab,
in your <code>$HOME</code> directory, and typically using your login shell. It is the ''script's responsibility'' to:
* Change to a suitable working directory.
* Recognize and use the other nodes of the job.
** This is generally done by directing an [http://en.wikipedia.org/wiki/Message_Passing_Interface MPI] program to start on the nodes listed in the job's <code>$PBS_NODEFILE</code>.
** A job is perfectly free to run more than one program, such as several MPI programs in sequence.
* Set or propagate environment variables as needed to other nodes.


=== Applications ===
=== Environment variables ===
We use the [http://modules.sourceforge.net/  environment-modules] package to manage user applications.
The handling of environment variables across the cores and nodes that a job uses varies with the capabilities and configuration choices of the various MPI implementations.
This is similar to places like NERSC or PNNL.
<!-- It is recommended not to rely on defaults, but rather to make expectations and requirements explicit. -->
The basic CNM-specific user environment is configured automatically in <code>/etc/profile.d/cnm.{sh,csh}</code>.


For now the only applications are the [[HPC/Carbon Cluster - Development tools| Development tools]].
* Variables usually are '''propagated core-to-core''' for all processes on the first node.
* Variables usually are '''not propagated node-to-node.'''
; Consequences:
* You must [[HPC/Software/Module Setup | '''configure modules and $PATH''']] in your <code>$HOME/.bashrc</code> file to help mpirun and other mechanisms locate your programs on each compute node. Otherwise mpirun will fail as soon as more than one node is involved, either by not finding the binary itself or some of its required shared libraries, or more insidiously, using different shared libraries.
* Both the variable <code>$PBS_NODEFILE</code> and the file it points to are only available on the first node.
* To propagate (export) environment variables, use mpirun/mpiexec. However, there is no standard or quasi-standard across MPI flavors.
** [http://www.open-mpi.org/doc/v1.4/man1/mpirun.1.php#sect17 OpenMPI's mpirun] selectively exports some variables and supports the <code>-x</code> flag to export additional variables.
** Intel's mpiexec/mpiexec.hydra offer <code>-envlist ''NAME,NAME,...''</code> instead. See Sec. 2.4 in the ''Reference Manual'' at [http://software.intel.com/en-us/articles/intel-mpi-library-documentation/].


<!-- OBSOLETE:
See also: [[HPC/Directories|directory layout]].
Admin note:  The master copy of these files resides in <code>mgmt{01,02}:/opt/teamhpc/node-skel/etc/profile.d</code> and is distributed by <code>~root/bin/skeldistrib</code>.
-->


== Submitting jobs to Moab/Torque ==
== Submitting and managing jobs ==
=== Job Script ===
A job is described to Torque by a ''job script,'' which in most cases is a shell script containing directives for Torque.
* [[HPC/Submitting and Managing Jobs/Example Job Script | '''Example Job Script''']].


=== Submission ===
To submit a job, use Torque's [http://www.adaptivecomputing.com/resources/docs/torque/4-1-4/Content/topics/commands/qsub.htm qsub] command on a login node.
   qsub [-A accountname] [options] ''jobfile''
   qsub [-A accountname] [options] ''jobfile''
For details on options:
To review qsub operation and options, consult the following:
   man qsub
   man qsub
   qsub --help
   qsub --help


More details are at the
; Advanced usage:
[http://www.clusterresources.com/products/torque/docs/ Torque Manual], in particular
* You ''can'' submit follow-up jobs from within a parent job script while it runs on a compute node, but it may be better to leverage [http://www.adaptivecomputing.com/resources/docs/torque/4-1-4/Content/topics/commands/qsub.htm#W job dependencies.]
the [http://www.clusterresources.com/torquedocs21/commands/qsub.shtml qsub man page].
* [http://www.adaptivecomputing.com/resources/docs/torque/4-1-4/Content/topics/2-jobs/requestingRes.htm Requesting Resources]
* Alternatively, use Moab's [http://www.adaptivecomputing.com/resources/docs/mwm/6-1-9/Content/commands/msub.html msub] command.


The single main queue is <code>batch</code> and need not be specified.  All job routing decisions are handled by the scheduler. In particular, short jobs are accommodated by a daily reserved node and by ''backfill'' scheduling, i.e. "waving forward" while a big job waits for full resources to become available.
=== Queues ===
See [[HPC/Submitting and Managing Jobs/Queues and Policies | Queues and Policies]].


=== Debug queue ===
=== Checking job status ===
For testing job processing and your job environment, use <code>qsub -q debug</code> or <code>#PBS -q debug</code>.
Query your jobs using commands similar to those on other HPC clusters.
The queue accepts jobs under the following conditions
You can use either [http://www.adaptivecomputing.com/resources/docs/torque/4-1-4/Content/topics/12-appendices/commandsOverview.htm Torque commands]
nodes <= 2
or [http://www.adaptivecomputing.com/resources/docs/mwm/6-1-9/Content/a.gcommandoverview.html Moab commands].
ppn <= 4
Both access the same information, but through different routes, and offer different display styles.
walltime <= 1:00:00


== Checking job status ==
To get the regular output format:
[http://www.adaptivecomputing.com/resources/docs/torque/4-1-4/Content/topics/commands/qstat.htm qstat] [-u $USER]
[http://www.adaptivecomputing.com/resources/docs/mwm/6-1-9/Content/commands/showq.html showq] [-u $USER]
To get an ''alternate'' format (e.g. showing job names):
qstat -a [-u $USER]
showq -n [-u $USER]
You can always use <code>-u $USER</code> to show only your jobs.


Use the command [http://www.clusterresources.com/torquedocs21/commands/qstat.shtml qstat] (from PBS) or [http://www.clusterresources.com/products/mwm/moabdocs/commands/showq.shtml showq] (from [http://www.clusterresources.com/products/mwm/moabdocs/a.gcommandoverview.shtml Moab]):
==== Getting extra information ====
Show the nodes where a job runs:
qstat -n [-1] ''jobnum''
Give ''full'' information such as submit arguments and run directories, and do not wrap long lines ("-1" is the digit "one"):
qstat -f [''jobnum''] [-1]
To diagnose problems with "stuck" jobs, use the <code>-s</code> option, or simply Moab's [http://www.adaptivecomputing.com/resources/docs/mwm/6-1-9/Content/commands/checkjob.html checkjob] command. The last few lines of output give ''past events'' of the job and its current status.
qstat -s ''jobnum''
checkjob -v ''jobnum''
To get estimates on when your job might start:
showstart [-e all] ''jobnum''


; qstat [-u $USER]:
==== Peeking at job output files ====
; showq [-u $USER]: regular output
Each job has associated stdin and stderr streams which are gathered on its first node and are copied back only at job end time, according to the <code>#PBS</code> directives -e, -o, or -j [eo|oe]. If the user does explicitly capture any output in the script (using shell redirections <code>"&gt;"</code>) it can be difficult to assess a job's progress.
; qstat -a:
; showq -n: ''alternate'' format (showing job names)


=== Getting extra information ===
To inspect these streams ''while the job is running'', and also to review a job's script, use the <code>qpeek</code> command:
; qstat -n [-1] ''jobnum'': Show nodes where a job runs.
qpeek ''jobnum''
; qstat -f [''jobnum''] [-1]: ''Full'' information such as submit arguments and run directories. The "-1" option disables wrapping for long output lines.
qpeek -eo ''jobnum''
; checkjob [-v] ''jobnum'': Get extended jobs status information – useful to diagnose problems with "stuck" jobs.
qpeek -s ''jobnum''
Read its manual page for full documentation:
qpeek --man
This command is a Carbon re-implementation of a Torque "contrib" script.


== Removing jobs ==
To inspect a job's processes running on its compute nodes:
To retract a queued job or terminate an already running job:
qpeek -p ''jobnum''
 
=== Changing jobs after submission ===
Use <code>[http://www.adaptivecomputing.com/resources/docs/torque/4-1-4/Content/topics/commands/qalter.htm qalter]</code> to modify attributes of a job.
Only attributes specified on the command line will be changed.
If any of the specified attributes cannot  be modified for any reason, ''none'' of that job's attributes will be modified.
qalter [-l ''resource''] […] ''jobnumber'' …
qalter -A ''accountspec'' ''jobnumber'' …
* Most attributes can be changed only ''before'' the job starts. Others, notably [http://www.adaptivecomputing.com/resources/docs/torque/4-1-4/Content/topics/commands/qsub.htm#-m mail options] and [http://www.adaptivecomputing.com/resources/docs/torque/4-1-4/Content/topics/commands/qsub.htm#-W job dependencies] can be meaningfully adjusted after start.
 
; Advanced usage:
* ''Extending'' the <code>walltime</code> resource of an already running job requires Torque administrator permission. You yourself can extend prior to running, and shorten always.  '''Warning:''' In either case, the new <code>walltime</code> limit will be measured from start, not as remaining time. If a new walltime limit is shorter than the elapsed time, the job will be killed right away.
* You cannot modify the job script itself after submission. You will have to remove and resubmit the job.
Contact me ([[User:Stern|stern]]) to arrange for interventions.
 
=== Removing jobs ===
To retract a queued job or kill an already running job:
   qdel ''jobnumber''
   qdel ''jobnumber''
[http://www.adaptivecomputing.com/resources/docs/torque/4-1-4/Content/topics/12-appendices/prologueAndEpliogueScripts.htm Torque epilogue scripts] will immediately remove remaining user processes and user files from the used compute nodes upon job termination, be it natural or after qdel.


== Example job file ==
== Declaring that a job requires licenses ==
A job that is to run software that requires a commercial license must declare such to the job scheduler (software that decides when and on which nodes the job could start).
The job scheduler monitors known licenses for availability, and so will know which and how many licenses are available at any given time.


* sample job file for Infiniband interconnect (recommended):
'''Note:''' License availability checks do not happen instantaneously but rather by polling, roughly once every minute.
<pre>
: This means that if you or another user happen to start a license-consuming application after the last check but before the relevant newly started job tries to access the same license, that job will terminate with a license failure.
#!/bin/bash


##  Basics: Number of nodes, processors per node (ppn), and walltime (hhh:mm:ss)
=== One license type ===
#PBS -l nodes=5:ppn=8
To request a single license token of a single type, use a <code>gres</code> ("general resource") directive:
#PBS -l walltime=0:10:00
#PBS -l '''gres'''=''license_name''
#PBS -N job_name
To request several tokens of a type, place either a <code>:</code> or <code>+</code> as separator, followed by the number of tokens required:
#PBS -A account
#PBS -l '''gres'''=''license_name:count''


## File names for stdout and stderr.  If not set here, the defaults
For compatibility, the resource type <code>software</code> is accepted, but ''not for multiple types'' of licenses.
## are <JOBNAME>.o<JOBNUM> and <JOBNAME>.e<JOBNUM>
#PBS -l '''software'''=''license_name''
#PBS -o job.out
#or
#PBS -e job.err
#PBS -l '''software'''=''license_name:count''


## send mail at begin, end, abort, or never (b, e, a, n)
=== Several license types ===
#PBS -m ea
To request several different licenses, use <code>gres</code> with list of license tokens and their optional counts separated by <code><font color="#1A1">'''%'''</font></code>:
#PBS -l gres=''license1_name''<font color="#1A1">'''%'''</font>''license2_name''
#PBS -l gres=''license1_name:count1''<font color="#1A1">'''%'''</font>''license2_name''


# change into the directory where qsub will be executed
The <code>software</code> resource ''cannot'' request multiple license types.
cd $PBS_O_WORKDIR
(If several <code>software</code> resources are requested, only the ''last one'' would be active.)


# count allocated cores
== Interactive node access ==
NPROCS=`wc -l < $PBS_NODEFILE`
There are two ways to interactively access nodes:
* Requesting an interactive job
qsub -I [-l nodes=…]
: This is a special Torque job submission that will wait for the requested nodes to become available, and then you will be dropped into a login shell on the first core of the first node. $PBS_NODEFILE is set as in regular (non-interactive) jobs.
* Side-access a node that already runs a job, such as to debug or inspect running processes or node-local files.
: You can use <code>ssh</code> to interactively access any compute node on which a job of yours is running (use <code>qsub -n ''jobnum''</code> to find out the node names). As soon as a node no longer runs at least ''one'' of your jobs, your ssh session to that node will be terminated. You may find the following commands helpful:
/usr/sbin/lsof -u $USER
/usr/sbin/lsof -c ''command_name''
/usr/sbin/lsof -p ''pid''
: - '''l'''i'''s'''t '''o'''pen '''f'''iles; [http://linux.die.net/man/8/lsof man page.]
ps -fu $USER
: - list your own processes.
psuser
: - a Carbon-specific wrapper for [http://linux.die.net/man/1/ps ps] to list user-only processes (yours and other regular users', excluding system processes).


# start MPI job over default interconnect
== Job notification mails ==
mpirun -machinefile $PBS_NODEFILE -np $NPROCS \
You can request a notification message to be sent to you when a job begins, ends, or aborts. The message is rather terse and looks like this:
        programname
'''Subject:''' PBS Job 281399.sched1.carboncluster
</pre>
* If your program reads from files or takes options and/or arguments, use and adjust one of the following forms
PBS Job Id: 281399.sched1.carboncluster
  mpirun -machinefile $PBS_NODEFILE -np $NPROCS \
Job Name:  test.job
        programname < run.in
Exec host:  n103/0
Execution terminated
Exit_status=0
resources_used.cput=00:03:00
resources_used.mem=123kb
  resources_used.vmem=240kb
  resources_used.walltime=00:03:04


mpirun -machinefile $PBS_NODEFILE -np $NPROCS \
In the above, Exit_status=0 reflects the shell exit status, where 0 means success.
        programname  -options arguments < run.in


  mpirun -machinefile $PBS_NODEFILE -np $NPROCS \
The following means your job was terminated because it ran beyond its allocated walltime:
        programname < run.in > run.out 2> run.err
  Exit_status=271
: The shell exit status is this number mod 256, in this case <code>271 % 256 = 15</code>, corresonding to <code>SIGTERM</code>.


:In this form, anything after <code>programname</code> is optional. If you use specific redirections for stdout or stderr as shown (&gt;, 2&gt;), the job-global files <code>job.out, job.err</code> declared earlier will remain empty or only contain output from your shell startup files (which should really be silent), and the rest of your job script.
== Recording where a job ran ==
* Infiniband (OpenIB) is the default (and fast) interconnect mechanism for MPI jobs. This is configured through the environment variable <code>$OMPI_MCA_btl</code>.
The [[#Job notification mails]] explained above do not include $PBS_O_WORKDIR. You will have to arrange for yourself to record that information.
* To select ethernet transport (e.g. for ''embarrasingly parallel'' jobs), specify an <code>-mca</code> option:
One way to do that would be to simply record all job numbers and their working directories at the beginning of each job:
<pre>
<source lang="bash">
mpirun -machinefile $PBS_NODEFILE -np $NPROCS \
#!/bin/bash
-mca btl self,tcp \
#PBS -l nodes=…
        programname
#PBS -l walltime=…
</pre>
#PBS -m bea


=== The ''account'' parameter ===
cd $PBS_O_WORKDIR
The parameter for option <code>-A account</code> is in most cases the CNM proposal, specified as follows:
echo $PBS_JOBID $PWD >> $HOME/jobdirs
; <code>cnm123</code>: (3 digits) for proposals below 1000
; <code>cnm01234</code>: (5 digits, 0-padded) for proposals from 1000 onwards.
; <code>user</code>: (the actual string "user", ''not'' your user name) for a limited personal startup allocation
; <code>staff</code>: for discretionary access by staff.


You can check your account balance in hours as follows:
  mybalance -h
</source>
gbalance -u $USER -h
To query a job's directory, run:
  grep ''jobnumber'' ~/jobdirs


== PPN Tricks ==
== Sending mail while a job is running ==
PBS will generate a <code>$PBS_NODEFILE</code> containing the name of each allocated node exactly ''ppn'' times, e.g. for a job with
It may be useful to have a job automatically "phone home" sometime after it started.
#PBS -l nodes=2:ppn=4
This can be accomplished at job submission time by arranging a mail to be sent, using a few added commands in the job script.
PBS will produce a nodefile like this:
n011
n011
n011
n011
n037
n037
n037
n037
Each ''Carbon'' node has 8 cores, and for many jobs users indeed request <code>ppn=8</code>.
However, you may need to request fewer cores, e.g. for the following reasons:
* your application is not parallelized
* your application has limited hardcoded parallelization, e.g. for 2 or 4 cores only (Carbon nodes have 8 cores)
* your application runs multi-threaded but uses <code>$PBS_NODEFILE</code> to infer the number of processes to start
* your application runs busy service processes or service threads (e.g. NWChem)
* your application saturates a resource, e.g. memory bandwidth (some large VASP calculations)
* the node's memory is not sufficient to run processes on all cores.
Select a remedy from the following scenarios.
=== Shared vs. Exclusive Node Access ===
; General node sharing:
: When a job requires only a few cores and a commensurate fraction of other resources, simply specify <code>ppn</code> as needed:
#PBS -l nodes=''nnn'':ppn=4
: In this case, the remaining cores may be allocated to other jobs, which is the ''default'' policy:
#PBS -l '''[http://www.clusterresources.com/products/mwm/13.3rmextensions.shtml#naccesspolicy naccesspolicy]=SHARED'''
; User-specific node sharing:
: To permit node sharing among your own jobs, specify a ''SINGLEUSER'' policy:
#PBS -l nodes=''nnn'':ppn=2
#PBS -l '''naccesspolicy=SINGLEUSER'''
; Exclusive node access:
: When your job requires only a few cores but a disproportionate fraction of another resource on a node (such as most of its memory or a lot of I/O bandwidth), claim the entire node:
#PBS -l nodes=''nnn'':ppn=4
#PBS '''-l naccesspolicy=SINGLEJOB'''
: PBS will reserve the entire node(s), but place each node name only <code>ppn</code> times in the <code>$PBS_NODEFILE</code>.


=== Different PPN by node ===
<syntaxhighlight lang="bash">
: When your first MPI process (the "master" process) requires more memory than your other "worker" processes, give several <code>nodes</code> specifications, separated by a <code>"+"</code> character (which is unusal and born of historical necessity):
#!/bin/bash
#PBS -l nodes=1:ppn=1+2:ppn=4
#PBS
#PBS -l naccesspolicy=SINGLEJOB
: For clarity, the <code>nodes</code> specification in this example reads as follows:
<font color="#888">nodes = ( 1:ppn=1 ) + ( 2:ppn=4 )</font>
: This will request 3 node exclusively, but the first node will occur only ''once'' in the <code>$PBS_NODEFILE</code>, e.g.
n011
n012
n012
n012
n012
n034
n034
n034
n034


In all of the preceding scenarios the following applies:
cd $PBS_O_WORKDIR
* The <code>$PBS_NODEFILE</code> seen by the job script will always match <code>ppn</code>.
* For accounting, the job will be ''billed'' by the number of cores blocked from use by other users, i.e., <code>ncores=ppn</code> for shared nodes, and <code>ncores=8</code> otherwise.


== Multithreading (OpenMP) ==
# arrange notification mail
When you wish to use multithreading, you must ensure that the total number of "busy" user threads and processes corresponds to the number of cores requested from PBS. Today, multithreading in applications and libraries is typically programmed using the [http://en.wikipedia.org/wiki/OpenMP OpenMP interface] and the number of threads is controlled by the environment variable <code>$OMP_NUM_THREADS</code>.
(
Select from the following scenarios.
    sleep 600 # time offset parameter, in seconds
    tail -20 somefile.out | mailx -s "$PBS_JOBID: partial log" $USER
) &


; OpenMP, single entire node:
# main work here
#PBS -l nodes=1:ppn=8
...
</syntaxhighlight>
cd $PBS_O_WORKDIR
export OMP_NUM_THREADS=8
...
; OpenMP, single node shared:
#PBS -l nodes=1:ppn=4
...
cd $PBS_O_WORKDIR
export OMP_NUM_THREADS=`uniq -c $PBS_NODEFILE | awk '{print $1; exit}'`
...
: Here, the default policy "SHARED" is in effect, and OMP_NUM_THREADS is set automatically by counting the number of times that the first node occurs in <code>$PBS_NODEFILE</code>. This will allow you to vary or override the ''nodes'' setting using "qsub -l nodes=…" without having to edit it twice in the job file.
; [http://www.google.com/search?hl=en&rls=en-us&q=MPI+OpenMP OpenMP/MPI hybrid]:
: Making efficient use of multithreading on multiple nodes which communicate over MPI is fairly involved and is subject to ongoing research. PBS requests are the same as in the preceding case (OpenMP only), but you must export <code>OMP_NUM_THREADS</code> to all MPI satellite nodes. Further, since <code>$PBS_NODEFILE</code> is typically used to identify remote nodes, you must ''thin out'' the file before using it as <code>machinefile</code>:
#!/bin/bash
#PBS -l nodes=''nnn'':ppn=8
...
MACHINEFILE=$PBS_NODEFILE
...
if [ ''multithreaded'' ]            # insert specific condition
then
    MACHINEFILE=machinefile
    sort -u $PBS_NODEFILE > $MACHINEFILE
fi
...
mpirun -x OMP_NUM_THREADS=`uniq -c $PBS_NODEFILE | awk '{print $1; exit}'` \
    -machinefile $MACHINEFILE \
    -np `wc -l &lt; $MACHINEFILE` \
   
The <code>-x</code> option is specific to OpenMPI; please consult the documentation to achieve the same behavior in other MPI implementations.


== Interactive node access ==
Explanation:
* The commands inside <code>( … ) &</code> run in a background track aside the main work.
* The first command, <code>sleep</code>, delays further action (in this track only) by the specified time in seconds.
** To emphasize minutes, write <code>sleep $(( 10 * 60 ))</code> .
** To set the time relative to the walltime limit, leverage the <code>$PBS_WALLTIME</code> variable, e.g. as in <code>sleep $(( PBS_WALLTIME - ( ''minutes'' * 60 ) )).</code>
* The <code>tail</code> command is a stand-in for any inspection command you might want to run. An even simpler command could be <code>ls -lt</code>. You could also use several commands enclosed by parentheses, <code>( cmd1; cmd2; ... )</code>, to construct a more elaborate mail body.
* The <code>mailx</code> command, following the pipe symbol <code>|</code>, reads the output of your inspection command and sends you a mail with the subject specified by the <code>-s</code> option.
* If the main part of the job finishes before the pre-mail sleep is over no mail will be sent.


* You can use <code>ssh</code> to interactively access any compute node on which a job of yours is running. As soon as a node no longer runs at least one of your jobs, your ssh session to that node will be terminated.
Note: The Moab scheduler also supports [http://www.adaptivecomputing.com/resources/docs/mwm/6-1-9/Content/topics/triggers/triggersIndex.html triggers], but they run in a different environment, not on an allocated node.


[[Category:HPC|Getting Started]]
[[Category:HPC|Getting Started]]

Latest revision as of 21:10, May 12, 2023

Processing flow of a typical job.

Introduction

To accept and run jobs, Carbon uses the Torque Resource Manager, which is a variant of PBS. A separate software component called a scheduler chooses the nodes and the start time for jobs according to the job parameters, resource availability, and configuration. Carbon uses the Moab Workload Manager as scheduler. The resources used by each job (primarily the CPU time) are tracked and managed by the Gold Allocation Manager.

Job processing

A job consists of a job script and applications called therein. The figure at the top shows schematically how a typical job is processed. Unless otherwise told, Torque will execute the job script on your behalf on the first node allocated by Moab, in your $HOME directory, and typically using your login shell. It is the script's responsibility to:

  • Change to a suitable working directory.
  • Recognize and use the other nodes of the job.
    • This is generally done by directing an MPI program to start on the nodes listed in the job's $PBS_NODEFILE.
    • A job is perfectly free to run more than one program, such as several MPI programs in sequence.
  • Set or propagate environment variables as needed to other nodes.

Environment variables

The handling of environment variables across the cores and nodes that a job uses varies with the capabilities and configuration choices of the various MPI implementations.

  • Variables usually are propagated core-to-core for all processes on the first node.
  • Variables usually are not propagated node-to-node.
Consequences
  • You must configure modules and $PATH in your $HOME/.bashrc file to help mpirun and other mechanisms locate your programs on each compute node. Otherwise mpirun will fail as soon as more than one node is involved, either by not finding the binary itself or some of its required shared libraries, or more insidiously, using different shared libraries.
  • Both the variable $PBS_NODEFILE and the file it points to are only available on the first node.
  • To propagate (export) environment variables, use mpirun/mpiexec. However, there is no standard or quasi-standard across MPI flavors.
    • OpenMPI's mpirun selectively exports some variables and supports the -x flag to export additional variables.
    • Intel's mpiexec/mpiexec.hydra offer -envlist NAME,NAME,... instead. See Sec. 2.4 in the Reference Manual at [1].

See also: directory layout.

Submitting and managing jobs

Job Script

A job is described to Torque by a job script, which in most cases is a shell script containing directives for Torque.

Submission

To submit a job, use Torque's qsub command on a login node.

 qsub [-A accountname] [options] jobfile

To review qsub operation and options, consult the following:

 man qsub
 qsub --help
Advanced usage
  • You can submit follow-up jobs from within a parent job script while it runs on a compute node, but it may be better to leverage job dependencies.
  • Requesting Resources
  • Alternatively, use Moab's msub command.

Queues

See Queues and Policies.

Checking job status

Query your jobs using commands similar to those on other HPC clusters. You can use either Torque commands or Moab commands. Both access the same information, but through different routes, and offer different display styles.

To get the regular output format:

qstat [-u $USER]
showq [-u $USER]

To get an alternate format (e.g. showing job names):

qstat -a [-u $USER]
showq -n [-u $USER]

You can always use -u $USER to show only your jobs.

Getting extra information

Show the nodes where a job runs:

qstat -n [-1] jobnum

Give full information such as submit arguments and run directories, and do not wrap long lines ("-1" is the digit "one"):

qstat -f [jobnum] [-1]

To diagnose problems with "stuck" jobs, use the -s option, or simply Moab's checkjob command. The last few lines of output give past events of the job and its current status.

qstat -s jobnum
checkjob -v jobnum

To get estimates on when your job might start:

showstart [-e all] jobnum

Peeking at job output files

Each job has associated stdin and stderr streams which are gathered on its first node and are copied back only at job end time, according to the #PBS directives -e, -o, or -j [eo|oe]. If the user does explicitly capture any output in the script (using shell redirections ">") it can be difficult to assess a job's progress.

To inspect these streams while the job is running, and also to review a job's script, use the qpeek command:

qpeek jobnum
qpeek -eo jobnum
qpeek -s jobnum

Read its manual page for full documentation:

qpeek --man

This command is a Carbon re-implementation of a Torque "contrib" script.

To inspect a job's processes running on its compute nodes:

qpeek -p jobnum

Changing jobs after submission

Use qalter to modify attributes of a job. Only attributes specified on the command line will be changed. If any of the specified attributes cannot be modified for any reason, none of that job's attributes will be modified.

qalter [-l resource] […] jobnumber …
qalter -A accountspec jobnumber
  • Most attributes can be changed only before the job starts. Others, notably mail options and job dependencies can be meaningfully adjusted after start.
Advanced usage
  • Extending the walltime resource of an already running job requires Torque administrator permission. You yourself can extend prior to running, and shorten always. Warning: In either case, the new walltime limit will be measured from start, not as remaining time. If a new walltime limit is shorter than the elapsed time, the job will be killed right away.
  • You cannot modify the job script itself after submission. You will have to remove and resubmit the job.

Contact me (stern) to arrange for interventions.

Removing jobs

To retract a queued job or kill an already running job:

 qdel jobnumber

Torque epilogue scripts will immediately remove remaining user processes and user files from the used compute nodes upon job termination, be it natural or after qdel.

Declaring that a job requires licenses

A job that is to run software that requires a commercial license must declare such to the job scheduler (software that decides when and on which nodes the job could start). The job scheduler monitors known licenses for availability, and so will know which and how many licenses are available at any given time.

Note: License availability checks do not happen instantaneously but rather by polling, roughly once every minute.

This means that if you or another user happen to start a license-consuming application after the last check but before the relevant newly started job tries to access the same license, that job will terminate with a license failure.

One license type

To request a single license token of a single type, use a gres ("general resource") directive:

#PBS -l gres=license_name

To request several tokens of a type, place either a : or + as separator, followed by the number of tokens required:

#PBS -l gres=license_name:count

For compatibility, the resource type software is accepted, but not for multiple types of licenses.

#PBS -l software=license_name
#or
#PBS -l software=license_name:count

Several license types

To request several different licenses, use gres with list of license tokens and their optional counts separated by %:

#PBS -l gres=license1_name%license2_name
#PBS -l gres=license1_name:count1%license2_name

The software resource cannot request multiple license types. (If several software resources are requested, only the last one would be active.)

Interactive node access

There are two ways to interactively access nodes:

  • Requesting an interactive job
qsub -I [-l nodes=…]
This is a special Torque job submission that will wait for the requested nodes to become available, and then you will be dropped into a login shell on the first core of the first node. $PBS_NODEFILE is set as in regular (non-interactive) jobs.
  • Side-access a node that already runs a job, such as to debug or inspect running processes or node-local files.
You can use ssh to interactively access any compute node on which a job of yours is running (use qsub -n jobnum to find out the node names). As soon as a node no longer runs at least one of your jobs, your ssh session to that node will be terminated. You may find the following commands helpful:
/usr/sbin/lsof -u $USER
/usr/sbin/lsof -c command_name
/usr/sbin/lsof -p pid
- list open files; man page.
ps -fu $USER
- list your own processes.
psuser
- a Carbon-specific wrapper for ps to list user-only processes (yours and other regular users', excluding system processes).

Job notification mails

You can request a notification message to be sent to you when a job begins, ends, or aborts. The message is rather terse and looks like this:

Subject: PBS Job 281399.sched1.carboncluster

PBS Job Id: 281399.sched1.carboncluster
Job Name:   test.job
Exec host:  n103/0
Execution terminated
Exit_status=0
resources_used.cput=00:03:00
resources_used.mem=123kb
resources_used.vmem=240kb
resources_used.walltime=00:03:04

In the above, Exit_status=0 reflects the shell exit status, where 0 means success.

The following means your job was terminated because it ran beyond its allocated walltime:

Exit_status=271
The shell exit status is this number mod 256, in this case 271 % 256 = 15, corresonding to SIGTERM.

Recording where a job ran

The #Job notification mails explained above do not include $PBS_O_WORKDIR. You will have to arrange for yourself to record that information. One way to do that would be to simply record all job numbers and their working directories at the beginning of each job:

#!/bin/bash
#PBS -l nodes=…
#PBS -l walltime=…
#PBS -m bea

cd $PBS_O_WORKDIR
echo $PBS_JOBID $PWD >> $HOME/jobdirs

…

To query a job's directory, run:

grep jobnumber ~/jobdirs

Sending mail while a job is running

It may be useful to have a job automatically "phone home" sometime after it started. This can be accomplished at job submission time by arranging a mail to be sent, using a few added commands in the job script.

#!/bin/bash
#PBS …

cd $PBS_O_WORKDIR

# arrange notification mail
(
    sleep 600	# time offset parameter, in seconds
    tail -20 somefile.out | mailx -s "$PBS_JOBID: partial log" $USER
) &

# main work here

Explanation:

  • The commands inside ( … ) & run in a background track aside the main work.
  • The first command, sleep, delays further action (in this track only) by the specified time in seconds.
    • To emphasize minutes, write sleep $(( 10 * 60 )) .
    • To set the time relative to the walltime limit, leverage the $PBS_WALLTIME variable, e.g. as in sleep $(( PBS_WALLTIME - ( minutes * 60 ) )).
  • The tail command is a stand-in for any inspection command you might want to run. An even simpler command could be ls -lt. You could also use several commands enclosed by parentheses, ( cmd1; cmd2; ... ), to construct a more elaborate mail body.
  • The mailx command, following the pipe symbol |, reads the output of your inspection command and sends you a mail with the subject specified by the -s option.
  • If the main part of the job finishes before the pre-mail sleep is over no mail will be sent.

Note: The Moab scheduler also supports triggers, but they run in a different environment, not on an allocated node.