View source for HPC/Submitting and Managing Jobs/Advanced node selection

== Node Types ==

=== Hardware ===
{{Template:Section node types}}

=== Selecting node types for jobs ===

Jobs are directed automatically onto nodes of the same generation, with preference for gen2.  Unless specifically requested, jobs will never mix generations. This will avoid disparate CPU speeds and MPI communication setup in a job.  You ''can'' force jobs onto either node set in the job script after <code>#PBS</code> or on the <code>qsub</code> command line by suffixing the <code>nodes=</code> specifier with a ''property'' such as <code>:gen6</code> or <code>:gen8</code>.  For example, to run on 2 nodes with 8 cores each:

<!--
<syntaxhighlight lang="bash">
qsub -l nodes=2:ppn=16:gen1  foo.job	# not recommended for VASP
qsub -l nodes=2:ppn=16:gen2  foo.job
</syntaxhighlight>
The following are (as of now) equivalent, since "bigmem" currently implies "gen2":
<syntaxhighlight lang="bash">
qsub -l nodes=2:ppn=16:gen2:bigmem  foo.job
qsub -l nodes=2:ppn=16:bigmem       foo.job
</syntaxhighlight>
-->

<!-- See also:
	http://www.clusterresources.com/torquedocs21/2.1jobsubmission.shtml#resources
// Redirects to an unrelated domain as of the mid-2010's.
-->

* [[HPC/Software/Catalog/vasp | '''Special note for VASP''']]

=== Recommendations ===
* Avoid specifying a node generation unless you have a reason. This widens the pool of nodes suitable to run your job, thus decreasing wait times.
* With <code>ppn=16</code> your jobs may run on any node generation.
<!--
* To get you high memory ''per core'', specify <code>:bigmem</code> (6 GB) or <code>:hugemem</code> (8 GB, charged at 2 × walltime rate).
* With <code>ppn=4</code> your jobs have a fairly good chance of running in the "attic" of gen3 nodes which are often busy with <code>ppn=8</code> jobs.
-->

== Advanced control of processeors-per-node (PPN) ==
Each ''Carbon'' node has nodes with different CPU core counts.
A request of <code>ppn=16</code> in the job submission can get routed onto a node with exactly 16 cores, or the job can get routed to a node with more cores, shareable by other jobs.

You can specify if your application is fine to run on ''shared'' nodes or needs nodes for ''exclusive'' use, such as for the following reasons:
* your application is not parallelized,
* your application has limited hardcoded parallelization, e.g. for 2 or 4 cores only,
* your application runs multi-threaded but uses <code>$PBS_NODEFILE</code> to infer the number of processes to start,
* your application runs busy service processes or service threads (e.g. NWChem),
* your application saturates a resource, e.g. memory bandwidth (some large VASP calculations),
* the node's memory is exhausted by fewer application processeses than there are cores available.
Depending on the reason, the node either may be or must ''not'' be used by other jobs.
In the past, the only way to achieve exclusive but undersubscribed node access was to request <code>ppn=8</code> and then to ''thin out'' a copy of the nodefile before passing it to the application.
To eliminate the need to edit the nodefile, use the <code>-l naccesspolicy=…</code> flag to differentiate between resources ''requested from Moab'' from those ''passed to the application'' (in $PBS_NODEFILE).
<!--
You can control the following aspects of your job:
; Resources requested from Moab:
: <code>-l nodes=''nnn'':ppn=''m''<nowiki>[</nowiki>:''feature''<nowiki>[</nowiki>:''feature''<nowiki>]]</nowiki></code>
: <code>-l naccesspolicy=''value''</code>
; Resources passed to the application:
:* <code>$PBS_NODEFILE</code> – for MPI
:* <code>$OMP_NUM_THREADS</code> – for OpenMP multithreading

As part of the job startup procedure PBS will generate a <code>$PBS_NODEFILE</code> containing the name of each allocated node exactly ''ppn'' times, e.g. for a job with
 #PBS -l nodes=2:ppn=4
PBS will produce a nodefile like this:
 n011
 n011
 n011
 n011
 n037
 n037
 n037
 n037
-->

Select an option from the following scenarios.
=== Shared vs. Exclusive Node Access ===
; Permit other users and jobs:
: When a job requires only a few cores and a commensurate fraction of other resources, simply specify <code>ppn</code> as needed:
 #PBS -l nodes=''nnn'':ppn=4
: In this case, the remaining cores may be allocated to other jobs, which is the ''default'' policy:
 #PBS -l '''[http://www.adaptivecomputing.com/resources/docs/mwm/5.3nodeaccess.php naccesspolicy]=SHARED'''
; Permit only ''your own'' jobs:
 #PBS -l nodes=''nnn'':ppn=2
 #PBS -l '''naccesspolicy=SINGLEUSER -n'''
; Permit only ''one'' job per node, no sharing:
: When your job requires only a few cores but a disproportionate fraction of another resource on a node (such as most of its memory or a lot of I/O bandwidth), claim the entire node:
 #PBS -l nodes=''nnn'':ppn=4
 #PBS '''-l naccesspolicy=SINGLEJOB -n'''
: PBS will reserve the entire node(s), but place each node name only <code>ppn</code> times in the <code>$PBS_NODEFILE</code>. This is also useful for MPI+OpenMP ("hybrid") programming, see below.
; Permit only one of ''your'' jobs, and permit other user's jobs:
 #PBS -l nodes=''nnn'':ppn=4
 #PBS '''-l naccesspolicy=UNIQUEUSER'''
: The node is shared, but limited to one job for any given user.

=== Memory constraints ===
Jobs that do not use all cores on a node are subject to memory contraints,
such that memory on each node is allocated in rough proportion to the number of cores requested.
Specifically, most of Carbon's current nodes have 2 physical memory banks (sets of DIMM slots),
and only ''one'' of these is made accessible to jobs that request <code>ppn ≤ ''n''<sub>proc</sub>/2</code>,
i.e., half the number of physical cores per node <code>''n''<sub>proc</sub></code>.

To request access to ''all'' memory on a node for jobs with <code>ppn &lt; ''n''<sub>proc</sub></code>
use the <code>qsub -n</code> option, as shown in the solutions above.
<!--
 managed by the ''cpusets'' feature introduced in TORQUE-3.

# Request ''entire'' nodes from TORQUE with <code>ppn=8</code>, then have MPI use fewer cores per node with the <code>mpirun -npernode ''M''</code> option (''M'' = 1 to 7), instead of the usual request for all cores in the nodefile, <code>-np $PPN_NP</code>.
 #PBS -l nodes=N:ppn=8:gen2
 ...
 my_ppn=3
 mpirun  -machinefile $PBS_NODEFILE -npernode $my_ppn \
 	foo-app ...
; Drawback: This solution is specific to OpenMPI.

See also: http://www.supercluster.org/pipermail/torqueusers/2013-March/015663.html
-->

=== Different PPN by node ===
: When your first MPI process (the "master" process) requires more memory than your other "worker" processes, give several <code>nodes</code> specifications, separated by a <code>"+"</code> character (which is unusal and born of historical necessity):
 #PBS -l nodes=1:ppn=1+2:ppn=4
 #PBS -l naccesspolicy=SINGLEJOB -n
: For clarity, the <code>nodes</code> specification in this example reads as follows:
 <font color="#888">nodes = ( 1:ppn=1 ) + ( 2:ppn=4 )</font>
: This will request 3 node exclusively, but the first node will occur only ''once'' in the <code>$PBS_NODEFILE</code>, e.g.
 n011
 n012
 n012
 n012
 n012
 n034
 n034
 n034
 n034

In all of the preceding scenarios the following applies:
* The <code>$PBS_NODEFILE</code> seen by the job script will always match <code>ppn</code>.
* For accounting, the job will be ''billed'' by the number of cores blocked from use by other users, i.e., <code>ncores=ppn</code> for shared nodes, and <code>ncores=8</code> otherwise.

== Multithreading using OpenMP ==
When you wish to use multithreading, you must ensure that the total number of "busy" user threads and processes corresponds to the number of cores requested from PBS. Today, multithreading in applications and libraries is typically programmed using the [http://en.wikipedia.org/wiki/OpenMP OpenMP interface] and the number of threads is controlled by the environment variable <code>$OMP_NUM_THREADS</code>.
Select from the following scenarios.

=== Pure OpenMP, single entire node ===
<syntaxhighlight lang="bash">
#PBS -l nodes=1:ppn=8
#PBS -l naccesspolicy=SINGLEJOB -n

cd $PBS_O_WORKDIR
export OMP_NUM_THREADS=$PBS_NUM_PPN
...
programname …
</syntaxhighlight>

=== Pure OpenMP, single node, possibly shared ===
Choose the number of cores <code>''n''</code> such that <code>1 ≤ ''n'' ≤ 8 (or 16)</code>:
<syntaxhighlight lang="bash">
#PBS -l nodes=1:ppn=n
...
cd $PBS_O_WORKDIR
export OMP_NUM_THREADS=$PBS_NUM_PPN
...
programname …
</syntaxhighlight>
Here, the default policy "SHARED" is in effect, and OMP_NUM_THREADS is set automatically by counting the number of times that the first node occurs in <code>$PBS_NODEFILE</code>. This will allow you to vary or override the ''nodes'' setting using "qsub -l nodes=…" without having to edit it twice in the job file.

=== OpenMP/MPI hybrid ===
Making efficient use of multithreading on multiple nodes which communicate over MPI is fairly involved and is subject to [http://www.google.com/search?hl=en&rls=en-us&q=MPI+OpenMP ongoing research]. Since <code>OMP_NUM_THREADS</code> is set to 1 by default on MPI satellite nodes, you must export this variable after you altered it in the job file.
<syntaxhighlight lang="bash">
#!/bin/bash
#PBS -l nodes=nnn:ppn=2
#PBS -l naccesspolicy=SINGLEJOB -n

# Calculate number of threads available per MPI process
cores_per_node=$( grep -c ^processor /proc/cpuinfo )
export OMP_NUM_THREADS=$(( cores_per_node / PBS_NUM_PPN ))

mpirun -x OMP_NUM_THREADS -machinefile  $PBS_NODEFILE -np $PBS_NP \
     programname …
</syntaxhighlight>
The <code>-x</code> option is specific to OpenMPI; please consult the documentation to achieve the same behavior in other MPI implementations.

The last example will ensure:
* you get allocated entire nodes (SINGLEJOB policy)
* you do not oversubscribe cores (OMP_NUM_THREADS is calculated from ppn)
* you only have one place to adjust (ppn), and can do so in the command line, or even post submission

It is assumed:
* The number of cores on the first node (running the job script) is the same as on the other nodes.
* All cores on a node will be used.

== Advanced: PBS_* Variables ==
The following environment variables are provided in a job environment, with their value being computed any job runtime.

<source lang="bash">
echo 'env | grep PBS_ | sort' | qsub -N inspect_env
# …
cat ~/inspect_env.o*
</source>
    '''PBS_JOBNAME'''=''jobname''				# from -N option, useful in job script for file names
 
    '''PBS_NP'''=1					# number of processors requested, '''automatically used for "mpirun -n …"'''
    '''PBS_NUM_NODES'''=1				# number of nodes, from resource "-l nodes=…"
    '''PBS_NUM_PPN'''=1				# ppn value, ditto
 
    '''PBS_NODEFILE'''=/var/spool/torque/aux/… 	# '''automatically used as "mpirun -machinefile …"''' option
    PBS_GPUFILE=/var/spool/torque/aux/…		# file containing GPUs to access, present only with "-l nodes=…:gpus=…"
    PBS_MICFILE=/var/spool/torque/aux/…		# (not applicable on Carbon)
 
    '''PBS_WALLTIME'''=3600				# requested (or default) walltime, in seconds
    '''PBS_ENVIRONMENT'''				# useful for detection in scripts. Usually "PBS_BATCH", but "PBS_INTERACTIVE" under "qsub -I"
 
    # PBS internals
    PBS_JOBCOOKIE=643…31D7
    PBS_JOBID=3141593.sched5.carboncluster
    PBS_MOMPORT=15003
    PBS_QUEUE=batch
    PBS_TASKNUM=1 
    PBS_NODENUM=0				# usually always 0 (even on daughter nodes!)
    PBS_VNODENUM=0				# ditto.
    PBS_VERSION=TORQUE-4.2.10
    PBS_O_QUEUE=batch
    PBS_O_SERVER=sched5
    PBS_O_SUBMIT_FILTER=/usr/local/sbin/torque_submitfilter
 
    # Unix-like basics
    PBS_O_HOME=/home/username
    PBS_O_HOST=n123.carboncluster
    PBS_O_LANG=C				# Unix locale(7)
    PBS_O_LOGNAME=username
    PBS_O_MAIL=/var/spool/mail/username
    PBS_O_PATH=/home/username/bin:…
    PBS_O_SHELL=/bin/bash
    PBS_O_WORKDIR=/home/username
<!--
$ strings /usr/sbin/pbs_mom | grep -E "^[A-Z]+_[A-Z_]{5,}$" | grep -Ev '^(A[UVW]|SIG|SUB|GLIB|CXX|MPI|WAIT|RADIX|NOT)' | sort | pr -w100 -Tv3
BATCH_ALLOC_COOKIE               MOAB_NODELIST                    PBS_NNODES
BATCH_PARTITION_ID               MOAB_PARTITION                   PBS_NODEFILE
BEOWULF_JOB_MAP                  MOAB_PROCCOUNT                   PBS_NODENUM
CLUSTER_ADDRS                    MOAB_TASKMAP                     PBS_NUM_NODES
CUDA_VISIBLE_DEVICES             OFFLOAD_DEVICES                  PBS_NUM_PPN
END_OF_LINE                      PBS_BATCH                        PBS_O_HOST
GET_TASKS                        PBS_BATCH_SERVICE_PORT           PBS_O_INITDIR
GPU_STATUS                       PBS_ENVIRONMENT                  PBS_O_ROOTDIR
JOIN_JOB_RADIX                   PBS_GPUFILE                      PBS_O_WORKDIR
KILL_JOB_RADIX                   PBS_INTERACTIVE                  PBS_QUEUE
MOAB_CLASS                       PBS_JOBCOOKIE                    PBS_RESOURCE_GRES
MOAB_GROUP                       PBS_JOBID                        PBS_RESOURCE_NODES
MOAB_JOBARRAYINDEX               PBS_JOBNAME                      PBS_SCHED_HINT
MOAB_JOBARRAYRANGE               PBS_MANAGER_SERVICE_PORT         PBS_TASKNUM
MOAB_JOBID                       PBS_MICFILE                      PBS_VERSION
MOAB_JOBNAME                     PBS_MOMPORT                      PBS_VNODENUM
MOAB_MACHINE                     PBS_MOM_SERVICE_PORT             PBS_WALLTIME
MOAB_NODECOUNT                   PBS_MSHOST
-->

[[Category:HPC]]