HPC/Submitting and Managing Jobs: Difference between revisions

From CNM Wiki
< HPC
Jump to navigation Jump to search
m (break out advanced sections to own page)
Line 5: Line 5:
See also: '''[[HPC/Directories|directory configuration]]'''.
See also: '''[[HPC/Directories|directory configuration]]'''.


== Submitting jobs to Moab/Torque ==
== Submitting and managing jobs ==
[[Image:HPC job flow.png|right|400px|thumb]]
[[Image:HPC job flow.png|right|400px|thumb]]
   qsub [-A accountname] [options] ''jobfile''
   qsub [-A accountname] [options] ''jobfile''
Line 25: Line 25:
  walltime <= 1:00:00
  walltime <= 1:00:00


== Checking job status ==
=== Checking job status ===
 
Use the command [http://www.clusterresources.com/torquedocs21/commands/qstat.shtml qstat] (from PBS) or [http://www.clusterresources.com/products/mwm/moabdocs/commands/showq.shtml showq] (from [http://www.clusterresources.com/products/mwm/moabdocs/a.gcommandoverview.shtml Moab]):
Use the command [http://www.clusterresources.com/torquedocs21/commands/qstat.shtml qstat] (from PBS) or [http://www.clusterresources.com/products/mwm/moabdocs/commands/showq.shtml showq] (from [http://www.clusterresources.com/products/mwm/moabdocs/a.gcommandoverview.shtml Moab]):


Line 34: Line 33:
; showq -n: ''alternate'' format (showing job names)
; showq -n: ''alternate'' format (showing job names)


=== Getting extra information ===
==== Getting extra information ====
; qstat -n [-1] ''jobnum'': Show nodes where a job runs.
; qstat -n [-1] ''jobnum'': Show nodes where a job runs.
; qstat -f [''jobnum''] [-1]: ''Full'' information such as submit arguments and run directories. The "-1" option disables wrapping for long output lines.
; qstat -f [''jobnum''] [-1]: ''Full'' information such as submit arguments and run directories. The "-1" option disables wrapping for long output lines.
; checkjob [-v] ''jobnum'': Get extended jobs status information – useful to diagnose problems with "stuck" jobs.
; checkjob [-v] ''jobnum'': Get extended jobs status information – useful to diagnose problems with "stuck" jobs.


== Removing jobs ==
=== Changing jobs after submission ===
You can use <code>the [http://www.adaptivecomputing.com/resources/docs/torque/commands/qalter.php qalter]</code> to modify the attributes of a job ''before it starts''.
Specify the attributes to be modified on the command line. All others will not be changed.
If any of the specified attributes cannot  be modified for any reason, ''none'' of that job's attributes will be modified.
<syntaxhighlight lang="bash">
qalter [-l resource] […] jobnumber
qalter -A accountspec jobnumber
</syntaxhighlight>
You cannot modify the job script itself after submission. You will have to remove and resubmit the job.
For exceptional situations, contact me ([[User:Stern|stern]]) to arrange for a manual intervention.
 
=== Removing jobs ===
To retract a queued job or terminate an already running job:
To retract a queued job or terminate an already running job:
   qdel ''jobnumber''
   qdel ''jobnumber''
Line 111: Line 121:
</syntaxhighlight>
</syntaxhighlight>


== Node Types ==
=== Advanced node selection ===
 
The PBS resource <code>-l nodes=…</code> can be refined further to:
=== Hardware ===
* Select specific node hardware generations
 
* Control shared vs. exclusive node access
Carbon has two major  [[HPC/Hardware Details | node types]], called '''gen1''' and '''gen2''', and gen2 is further differentiated by the amount of memory.
* Vary PPN within the job
 
* Accommodate multithreading (OpenMP).
{{Template:Table of node types}}
See [[HPC/Submitting Jobs/Advanced node selection]] for these topics.
 
Benchmarks show that gen2 nodes are about twice as fast as gen1 nodes for memory-intensive applications. (The X5300 series is hampered by a memory bandwidth bottleneck when all 8 cores are active and frequently access memory.)  Thus, gen1 nodes are charged at a discounted rate of 50% of the walltime actually used.
 
=== Selecting node types for jobs ===
 
Jobs are directed automatically onto either gen1 or gen2 nodes, with preference for gen2 if both are available.  Unless specifically requested, jobs will never mix generations. This will avoid disparate CPU speeds and MPI communication setup in a job.  You ''can'' force jobs onto either node set in the job script after <code>#PBS</code> or on the <code>qsub</code> command line by suffixing the <code>nodes=</code> specifier with a ''property'' such as <code>:gen1</code> or <code>:gen2</code>.  For example, to run on 2 nodes with 8 cores each:
 
<syntaxhighlight lang="bash">
qsub -l nodes=2:ppn=8:gen1  foo.job # not recommended for VASP
qsub -l nodes=2:ppn=8:gen2  foo.job
</syntaxhighlight>
 
The following are (as of now) equivalent, since "bigmem" currently implies "gen2":
 
<syntaxhighlight lang="bash">
qsub -l nodes=2:ppn=8:gen2:bigmem  foo.job
qsub -l nodes=2:ppn=8:bigmem      foo.job
</syntaxhighlight>
 
See also:
http://www.clusterresources.com/torquedocs21/2.1jobsubmission.shtml#resources
 
* [[HPC/Software/Catalog/vasp | '''Special note for VASP''']]
 
== PPN Tricks ==
Each ''Carbon'' node has 8 cores, and for many jobs users indeed request entire nodes by specififying <code>ppn=8</code> in the job submission.
However, you may need to request fewer cores, e.g. for the following reasons:
* your application is not parallelized,
* your application has limited hardcoded parallelization, e.g. for 2 or 4 cores only,
* your application runs multi-threaded but uses <code>$PBS_NODEFILE</code> to infer the number of processes to start,
* your application runs busy service processes or service threads (e.g. NWChem),
* your application saturates a resource, e.g. memory bandwidth (some large VASP calculations),
* the node's memory is exhausted by fewer application processeses than there are cores available.
Depending on the reason, the node either may be or must ''not'' be used by other jobs.
In the past, the only way to achieve exclusive but undersubscribed node access was to request <code>ppn=8</code> and then to ''thin out'' a copy of the nodefile before passing it to the application.
To eliminate the need to edit the nodefile, use the <code>-l naccesspolicy=…</code> flag to differentiate between resources ''requested from Moab'' from those ''passed to the application'' (in $PBS_NODEFILE).
<!--
You can control the following aspects of your job:
; Resources requested from Moab:
: <code>-l nodes=''nnn'':ppn=''m''<nowiki>[</nowiki>:''feature''<nowiki>[</nowiki>:''feature''<nowiki>]]</nowiki></code>
: <code>-l naccesspolicy=''value''</code>
; Resources passed to the application:
:* <code>$PBS_NODEFILE</code> – for MPI
:* <code>$OMP_NUM_THREADS</code> – for OpenMP multithreading
 
As part of the job startup procedure PBS will generate a <code>$PBS_NODEFILE</code> containing the name of each allocated node exactly ''ppn'' times, e.g. for a job with
#PBS -l nodes=2:ppn=4
PBS will produce a nodefile like this:
n011
n011
n011
n011
n037
n037
n037
n037
-->
 
Select an option from the following scenarios.
=== Shared vs. Exclusive Node Access ===
; Permit other users and jobs:
: When a job requires only a few cores and a commensurate fraction of other resources, simply specify <code>ppn</code> as needed:
#PBS -l nodes=''nnn'':ppn=4
: In this case, the remaining cores may be allocated to other jobs, which is the ''default'' policy:
#PBS -l '''[http://www.adaptivecomputing.com/resources/docs/mwm/5.3nodeaccess.php naccesspolicy]=SHARED'''
; Permit only your own jobs:
#PBS -l nodes=''nnn'':ppn=2
#PBS -l '''naccesspolicy=SINGLEUSER'''
; Permit only ''one'' job per node, no sharing:
: When your job requires only a few cores but a disproportionate fraction of another resource on a node (such as most of its memory or a lot of I/O bandwidth), claim the entire node:
#PBS -l nodes=''nnn'':ppn=4
#PBS '''-l naccesspolicy=SINGLEJOB'''
: PBS will reserve the entire node(s), but place each node name only <code>ppn</code> times in the <code>$PBS_NODEFILE</code>. This is also useful for MPI+OpenMP ("hybrid") programming, see below.
; Permit only one of ''your'' jobs, and permit other user's jobs:
#PBS -l nodes=''nnn'':ppn=4
#PBS '''-l naccesspolicy=UNIQUEUSER'''
: The node is shared, but limited to one job for any given user.
 
=== Different PPN by node ===
: When your first MPI process (the "master" process) requires more memory than your other "worker" processes, give several <code>nodes</code> specifications, separated by a <code>"+"</code> character (which is unusal and born of historical necessity):
#PBS -l nodes=1:ppn=1+2:ppn=4
#PBS -l naccesspolicy=SINGLEJOB
: For clarity, the <code>nodes</code> specification in this example reads as follows:
<font color="#888">nodes = ( 1:ppn=1 ) + ( 2:ppn=4 )</font>
: This will request 3 node exclusively, but the first node will occur only ''once'' in the <code>$PBS_NODEFILE</code>, e.g.
n011
n012
n012
n012
n012
n034
n034
n034
n034
 
In all of the preceding scenarios the following applies:
* The <code>$PBS_NODEFILE</code> seen by the job script will always match <code>ppn</code>.
* For accounting, the job will be ''billed'' by the number of cores blocked from use by other users, i.e., <code>ncores=ppn</code> for shared nodes, and <code>ncores=8</code> otherwise.
 
== Multithreading (OpenMP) ==
When you wish to use multithreading, you must ensure that the total number of "busy" user threads and processes corresponds to the number of cores requested from PBS. Today, multithreading in applications and libraries is typically programmed using the [http://en.wikipedia.org/wiki/OpenMP OpenMP interface] and the number of threads is controlled by the environment variable <code>$OMP_NUM_THREADS</code>.
Select from the following scenarios.
 
; Pure OpenMP, single entire node:
<syntaxhighlight lang="bash">
#PBS -l nodes=1:ppn=8
 
cd $PBS_O_WORKDIR
export OMP_NUM_THREADS=8
...
</syntaxhighlight>
; Pure OpenMP, single node, possibly shared:
: choose the number of cores <code>''n''</code> such that <code>1 ≤ ''n'' ≤ 8</code>:
<syntaxhighlight lang="bash">
#PBS -l nodes=1:ppn=''n''
...
cd $PBS_O_WORKDIR
export OMP_NUM_THREADS=`uniq -c $PBS_NODEFILE | awk '{print $1; exit}'`
...
</syntaxhighlight>
: Here, the default policy "SHARED" is in effect, and OMP_NUM_THREADS is set automatically by counting the number of times that the first node occurs in <code>$PBS_NODEFILE</code>. This will allow you to vary or override the ''nodes'' setting using "qsub -l nodes=…" without having to edit it twice in the job file.
; [http://www.google.com/search?hl=en&rls=en-us&q=MPI+OpenMP OpenMP/MPI hybrid]:
: Making efficient use of multithreading on multiple nodes which communicate over MPI is fairly involved and is subject to ongoing research. Since <code>OMP_NUM_THREADS</code> is set to 1 by default on MPI satellite nodes, you must export this variable after you altered it in the job file.
<syntaxhighlight lang="bash">
#!/bin/bash
#PBS -l nodes=''nnn'':ppn=4
#PBS -l naccesspolicy=SINGLEJOB
 
# Calculate number of threads available per MPI process
ppn_mpi=$( uniq -c $PBS_NODEFILE | awk '{print $1; exit}' )
ppn_phys=$( grep -c ^processor /proc/cpuinfo )
OMP_NUM_THREADS=$(( ppn_phys / ppn_mpi ))
 
mpirun -x OMP_NUM_THREADS \
    -machinefile  $PBS_NODEFILE \
    -np $(wc -l < $PBS_NODEFILE) \
    …
</syntaxhighlight>
The <code>-x</code> option is specific to OpenMPI; please consult the documentation to achieve the same behavior in other MPI implementations.
 
The last example will ensure:
* you get allocated entire nodes (SINGLEJOB policy)
* you do not oversubscribe cores (OMP_NUM_THREADS is calculated from ppn)
* you only have one place to adjust (ppn), and can do so in the command line, or even post submission
 
It is assumed:
* The number of cores on the first node (running the job script) is the same as on the other nodes.


== Interactive node access ==
== Interactive node access ==

Revision as of 21:00, May 23, 2011

User environment

We use the environment-modules package to manage user applications. Learn more at HPC/Software/Environment.

See also: directory configuration.

Submitting and managing jobs

HPC job flow.png 
 qsub [-A accountname] [options] jobfile

For details on options: 

 man qsub
 qsub --help

More details are at the Torque Manual, in particular the qsub man page.

The single main queue is batch and need not be specified. All job routing decisions are handled by the scheduler. In particular, short jobs are accommodated by a daily reserved node and by backfill scheduling, i.e. "waving forward" while a big job waits for full resources to become available.

Debug queue

For testing job processing and your job environment, use qsub -q debug or #PBS -q debug. The queue accepts jobs under the following conditions 

nodes <= 2
ppn <= 4
walltime <= 1:00:00

Checking job status

Use the command qstat (from PBS) or showq (from Moab):

qstat [-u $USER]
showq [-u $USER]
regular output
qstat -a
showq -n
alternate format (showing job names)

Getting extra information

qstat -n [-1] jobnum
Show nodes where a job runs.
qstat -f [jobnum] [-1]
Full information such as submit arguments and run directories. The "-1" option disables wrapping for long output lines.
checkjob [-v] jobnum
Get extended jobs status information – useful to diagnose problems with "stuck" jobs.

Changing jobs after submission

You can use the qalter to modify the attributes of a job before it starts. Specify the attributes to be modified on the command line. All others will not be changed. If any of the specified attributes cannot be modified for any reason, none of that job's attributes will be modified. 

qalter [-l resource] […] jobnumber
qalter -A accountspec jobnumber

You cannot modify the job script itself after submission. You will have to remove and resubmit the job. For exceptional situations, contact me (stern) to arrange for a manual intervention.

Removing jobs

To retract a queued job or terminate an already running job: 

 qdel jobnumber

Example job file

  • sample job file for Infiniband interconnect (recommended):
#!/bin/bash

##  Basics: Number of nodes, processors per node (ppn), and walltime (hhh:mm:ss)
#PBS -l nodes=5:ppn=8
#PBS -l walltime=0:10:00
#PBS -N job_name
#PBS -A account

## File names for stdout and stderr.  If not set here, the defaults
## are <JOBNAME>.o<JOBNUM> and <JOBNAME>.e<JOBNUM>
#PBS -o job.out
#PBS -e job.err

## send mail at begin, end, abort, or never (b, e, a, n)
#PBS -m ea

# change into the directory where qsub will be executed
cd $PBS_O_WORKDIR

# count allocated cores
NPROCS=`wc -l < $PBS_NODEFILE`

# start MPI job over default interconnect
mpirun -machinefile $PBS_NODEFILE -np $NPROCS \
        programname
  • If your program reads from files or takes options and/or arguments, use and adjust one of the following forms
mpirun -machinefile $PBS_NODEFILE -np $NPROCS \
       programname  < run.in

  

mpirun -machinefile $PBS_NODEFILE -np $NPROCS \
       programname  -options arguments < run.in

  

mpirun -machinefile $PBS_NODEFILE -np $NPROCS \
       programname < run.in > run.out 2> run.err
In this form, anything after programname is optional. If you use specific redirections for stdout or stderr as shown (>, 2>), the job-global files job.out, job.err declared earlier will remain empty or only contain output from your shell startup files (which should really be silent), and the rest of your job script.
  • Infiniband (OpenIB) is the default (and fast) interconnect mechanism for MPI jobs. This is configured through the environment variable $OMPI_MCA_btl.
  • To select ethernet transport (e.g. for embarrasingly parallel jobs), specify an -mca option:
mpirun -machinefile $PBS_NODEFILE -np $NPROCS \
	-mca btl self,tcp \
        programname

The account parameter

The parameter for option -A account is in most cases the CNM proposal, specified as follows:

cnm123
(3 digits) for proposals below 1000
cnm01234
(5 digits, 0-padded) for proposals from 1000 onwards.
user
(the actual string "user", not your user name) for a limited personal startup allocation
staff
for discretionary access by staff.

You can check your account balance in hours as follows: 

mybalance -h
gbalance -u $USER -h

Advanced node selection

The PBS resource -l nodes=… can be refined further to:

  • Select specific node hardware generations
  • Control shared vs. exclusive node access
  • Vary PPN within the job
  • Accommodate multithreading (OpenMP).

See HPC/Submitting Jobs/Advanced node selection for these topics.

Interactive node access

  • You can use ssh to interactively access any compute node on which a job of yours is running. As soon as a node no longer runs at least one of your jobs, your ssh session to that node will be terminated.
Retrieved from "https://wiki.anl.gov/wiki_cnm/index.php?title=HPC/Submitting_and_Managing_Jobs&oldid=3079"