HPC/Submitting and Managing Jobs/Advanced node selection: Difference between revisions

From CNM Wiki
Jump to navigation Jump to search
mNo edit summary
 
(50 intermediate revisions by the same user not shown)
Line 2: Line 2:


=== Hardware ===
=== Hardware ===
 
{{Template:Section node types}}
Carbon has two major  [[HPC/Hardware Details | node types]], called '''gen1''' and '''gen2''', and gen2 is further differentiated by the amount of memory.
 
{{Template:Table of node types}}
 
[[HPC/Benchmarks/Generation 1 vs 2|Benchmarks]] show that gen2 nodes are about twice as fast as gen1 nodes for memory-intensive applications. (The X5300 series is hampered by a memory bandwidth bottleneck when all 8 cores are active and frequently access memory.)  Thus, gen1 nodes are charged at a discounted rate of the walltime actually used. The discount also encourages continued productive use of gen1 nodes.


=== Selecting node types for jobs ===
=== Selecting node types for jobs ===


Jobs are directed automatically onto either gen1 or gen2 nodes, with preference for gen2 if both are available.  Unless specifically requested, jobs will never mix generations. This will avoid disparate CPU speeds and MPI communication setup in a job.  You ''can'' force jobs onto either node set in the job script after <code>#PBS</code> or on the <code>qsub</code> command line by suffixing the <code>nodes=</code> specifier with a ''property'' such as <code>:gen1</code> or <code>:gen2</code>.  For example, to run on 2 nodes with 8 cores each:
Jobs are directed automatically onto nodes of the same generation, with preference for gen2.  Unless specifically requested, jobs will never mix generations. This will avoid disparate CPU speeds and MPI communication setup in a job.  You ''can'' force jobs onto either node set in the job script after <code>#PBS</code> or on the <code>qsub</code> command line by suffixing the <code>nodes=</code> specifier with a ''property'' such as <code>:gen6</code> or <code>:gen8</code>.  For example, to run on 2 nodes with 8 cores each:


<!--
<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
qsub -l nodes=2:ppn=8:gen1  foo.job # not recommended for VASP
qsub -l nodes=2:ppn=16:gen1  foo.job # not recommended for VASP
qsub -l nodes=2:ppn=8:gen2  foo.job
qsub -l nodes=2:ppn=16:gen2  foo.job
</syntaxhighlight>
</syntaxhighlight>
The following are (as of now) equivalent, since "bigmem" currently implies "gen2":
The following are (as of now) equivalent, since "bigmem" currently implies "gen2":
<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
qsub -l nodes=2:ppn=8:gen2:bigmem  foo.job
qsub -l nodes=2:ppn=16:gen2:bigmem  foo.job
qsub -l nodes=2:ppn=8:bigmem      foo.job
qsub -l nodes=2:ppn=16:bigmem      foo.job
</syntaxhighlight>
</syntaxhighlight>
-->


See also:
<!-- See also:
http://www.clusterresources.com/torquedocs21/2.1jobsubmission.shtml#resources
http://www.clusterresources.com/torquedocs21/2.1jobsubmission.shtml#resources
// Redirects to an unrelated domain as of the mid-2010's.
-->


* [[HPC/Software/Catalog/vasp | '''Special note for VASP''']]
* [[HPC/Software/Catalog/vasp | '''Special note for VASP''']]


== PPN Tricks ==
=== Recommendations ===
Each ''Carbon'' node has 8 cores, and for many jobs users indeed request entire nodes by specififying <code>ppn=8</code> in the job submission.
* Avoid specifying a node generation unless you have a reason. This widens the pool of nodes suitable to run your job, thus decreasing wait times.
However, you may need to request fewer cores, e.g. for the following reasons:
* With <code>ppn=16</code> your jobs may run on any node generation.
<!--
* To get you high memory ''per core'', specify <code>:bigmem</code> (6 GB) or <code>:hugemem</code> (8 GB, charged at 2 × walltime rate).
* With <code>ppn=4</code> your jobs have a fairly good chance of running in the "attic" of gen3 nodes which are often busy with <code>ppn=8</code> jobs.
-->
 
== Advanced control of processeors-per-node (PPN) ==
Each ''Carbon'' node has nodes with different CPU core counts.
A request of <code>ppn=16</code> in the job submission can get routed onto a node with exactly 16 cores, or the job can get routed to a node with more cores, shareable by other jobs.
 
You can specify if your application is fine to run on ''shared'' nodes or needs nodes for ''exclusive'' use, such as for the following reasons:
* your application is not parallelized,
* your application is not parallelized,
* your application has limited hardcoded parallelization, e.g. for 2 or 4 cores only,
* your application has limited hardcoded parallelization, e.g. for 2 or 4 cores only,
Line 71: Line 78:
: In this case, the remaining cores may be allocated to other jobs, which is the ''default'' policy:
: In this case, the remaining cores may be allocated to other jobs, which is the ''default'' policy:
  #PBS -l '''[http://www.adaptivecomputing.com/resources/docs/mwm/5.3nodeaccess.php naccesspolicy]=SHARED'''
  #PBS -l '''[http://www.adaptivecomputing.com/resources/docs/mwm/5.3nodeaccess.php naccesspolicy]=SHARED'''
; Permit only your own jobs:
; Permit only ''your own'' jobs:
  #PBS -l nodes=''nnn'':ppn=2
  #PBS -l nodes=''nnn'':ppn=2
  #PBS -l '''naccesspolicy=SINGLEUSER'''
  #PBS -l '''naccesspolicy=SINGLEUSER -n'''
; Permit only ''one'' job per node, no sharing:
; Permit only ''one'' job per node, no sharing:
: When your job requires only a few cores but a disproportionate fraction of another resource on a node (such as most of its memory or a lot of I/O bandwidth), claim the entire node:
: When your job requires only a few cores but a disproportionate fraction of another resource on a node (such as most of its memory or a lot of I/O bandwidth), claim the entire node:
  #PBS -l nodes=''nnn'':ppn=4
  #PBS -l nodes=''nnn'':ppn=4
  #PBS '''-l naccesspolicy=SINGLEJOB'''
  #PBS '''-l naccesspolicy=SINGLEJOB -n'''
: PBS will reserve the entire node(s), but place each node name only <code>ppn</code> times in the <code>$PBS_NODEFILE</code>. This is also useful for MPI+OpenMP ("hybrid") programming, see below.
: PBS will reserve the entire node(s), but place each node name only <code>ppn</code> times in the <code>$PBS_NODEFILE</code>. This is also useful for MPI+OpenMP ("hybrid") programming, see below.
; Permit only one of ''your'' jobs, and permit other user's jobs:
; Permit only one of ''your'' jobs, and permit other user's jobs:
Line 83: Line 90:
  #PBS '''-l naccesspolicy=UNIQUEUSER'''
  #PBS '''-l naccesspolicy=UNIQUEUSER'''
: The node is shared, but limited to one job for any given user.
: The node is shared, but limited to one job for any given user.
=== Memory constraints ===
Jobs that do not use all cores on a node are subject to memory contraints,
such that memory on each node is allocated in rough proportion to the number of cores requested.
Specifically, most of Carbon's current nodes have 2 physical memory banks (sets of DIMM slots),
and only ''one'' of these is made accessible to jobs that request <code>ppn ≤ ''n''<sub>proc</sub>/2</code>,
i.e., half the number of physical cores per node <code>''n''<sub>proc</sub></code>.
To request access to ''all'' memory on a node for jobs with <code>ppn &lt; ''n''<sub>proc</sub></code>
use the <code>qsub -n</code> option, as shown in the solutions above.
<!--
managed by the ''cpusets'' feature introduced in TORQUE-3.
# Request ''entire'' nodes from TORQUE with <code>ppn=8</code>, then have MPI use fewer cores per node with the <code>mpirun -npernode ''M''</code> option (''M'' = 1 to 7), instead of the usual request for all cores in the nodefile, <code>-np $PPN_NP</code>.
#PBS -l nodes=N:ppn=8:gen2
...
my_ppn=3
mpirun  -machinefile $PBS_NODEFILE -npernode $my_ppn \
foo-app ...
; Drawback: This solution is specific to OpenMPI.
See also: http://www.supercluster.org/pipermail/torqueusers/2013-March/015663.html
-->


=== Different PPN by node ===
=== Different PPN by node ===
: When your first MPI process (the "master" process) requires more memory than your other "worker" processes, give several <code>nodes</code> specifications, separated by a <code>"+"</code> character (which is unusal and born of historical necessity):
: When your first MPI process (the "master" process) requires more memory than your other "worker" processes, give several <code>nodes</code> specifications, separated by a <code>"+"</code> character (which is unusal and born of historical necessity):
  #PBS -l nodes=1:ppn=1+2:ppn=4
  #PBS -l nodes=1:ppn=1+2:ppn=4
  #PBS -l naccesspolicy=SINGLEJOB
  #PBS -l naccesspolicy=SINGLEJOB -n
: For clarity, the <code>nodes</code> specification in this example reads as follows:
: For clarity, the <code>nodes</code> specification in this example reads as follows:
  <font color="#888">nodes = ( 1:ppn=1 ) + ( 2:ppn=4 )</font>
  <font color="#888">nodes = ( 1:ppn=1 ) + ( 2:ppn=4 )</font>
Line 105: Line 135:
* For accounting, the job will be ''billed'' by the number of cores blocked from use by other users, i.e., <code>ncores=ppn</code> for shared nodes, and <code>ncores=8</code> otherwise.
* For accounting, the job will be ''billed'' by the number of cores blocked from use by other users, i.e., <code>ncores=ppn</code> for shared nodes, and <code>ncores=8</code> otherwise.


== Multithreading (OpenMP) ==
== Multithreading using OpenMP ==
When you wish to use multithreading, you must ensure that the total number of "busy" user threads and processes corresponds to the number of cores requested from PBS. Today, multithreading in applications and libraries is typically programmed using the [http://en.wikipedia.org/wiki/OpenMP OpenMP interface] and the number of threads is controlled by the environment variable <code>$OMP_NUM_THREADS</code>.
When you wish to use multithreading, you must ensure that the total number of "busy" user threads and processes corresponds to the number of cores requested from PBS. Today, multithreading in applications and libraries is typically programmed using the [http://en.wikipedia.org/wiki/OpenMP OpenMP interface] and the number of threads is controlled by the environment variable <code>$OMP_NUM_THREADS</code>.
Select from the following scenarios.
Select from the following scenarios.


; Pure OpenMP, single entire node:
=== Pure OpenMP, single entire node ===
<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
#PBS -l nodes=1:ppn=8
#PBS -l nodes=1:ppn=8
#PBS -l naccesspolicy=SINGLEJOB -n


cd $PBS_O_WORKDIR
cd $PBS_O_WORKDIR
export OMP_NUM_THREADS=8
export OMP_NUM_THREADS=$PBS_NUM_PPN
...
...
programname …
</syntaxhighlight>
</syntaxhighlight>
; Pure OpenMP, single node, possibly shared:
 
: choose the number of cores <code>''n''</code> such that <code>1 ≤ ''n'' ≤ 8</code>:
=== Pure OpenMP, single node, possibly shared ===
Choose the number of cores <code>''n''</code> such that <code>1 ≤ ''n'' ≤ 8 (or 16)</code>:
<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
#PBS -l nodes=1:ppn=n
#PBS -l nodes=1:ppn=n
...
...
cd $PBS_O_WORKDIR
cd $PBS_O_WORKDIR
export OMP_NUM_THREADS=`uniq -c $PBS_NODEFILE | awk '{print $1; exit}'`
export OMP_NUM_THREADS=$PBS_NUM_PPN
...
...
programname …
</syntaxhighlight>
</syntaxhighlight>
: Here, the default policy "SHARED" is in effect, and OMP_NUM_THREADS is set automatically by counting the number of times that the first node occurs in <code>$PBS_NODEFILE</code>. This will allow you to vary or override the ''nodes'' setting using "qsub -l nodes=…" without having to edit it twice in the job file.
Here, the default policy "SHARED" is in effect, and OMP_NUM_THREADS is set automatically by counting the number of times that the first node occurs in <code>$PBS_NODEFILE</code>. This will allow you to vary or override the ''nodes'' setting using "qsub -l nodes=…" without having to edit it twice in the job file.
; [http://www.google.com/search?hl=en&rls=en-us&q=MPI+OpenMP OpenMP/MPI hybrid]:
 
: Making efficient use of multithreading on multiple nodes which communicate over MPI is fairly involved and is subject to ongoing research. Since <code>OMP_NUM_THREADS</code> is set to 1 by default on MPI satellite nodes, you must export this variable after you altered it in the job file.
=== OpenMP/MPI hybrid ===
Making efficient use of multithreading on multiple nodes which communicate over MPI is fairly involved and is subject to [http://www.google.com/search?hl=en&rls=en-us&q=MPI+OpenMP ongoing research]. Since <code>OMP_NUM_THREADS</code> is set to 1 by default on MPI satellite nodes, you must export this variable after you altered it in the job file.
<syntaxhighlight lang="bash">
<syntaxhighlight lang="bash">
#!/bin/bash
#!/bin/bash
#PBS -l nodes=nnn:ppn=4
#PBS -l nodes=nnn:ppn=2
#PBS -l naccesspolicy=SINGLEJOB
#PBS -l naccesspolicy=SINGLEJOB -n


ppn_mpi=$( uniq -c $PBS_NODEFILE | awk '{print $1; exit}' )    # grab first (and usually only) ppn from qsub
ppn_phys=$( grep -c ^processor /proc/cpuinfo )
# Calculate number of threads available per MPI process
# Calculate number of threads available per MPI process
OMP_NUM_THREADS=$(( ppn_phys / ppn_mpi ))
cores_per_node=$( grep -c ^processor /proc/cpuinfo )
export OMP_NUM_THREADS=$(( cores_per_node / PBS_NUM_PPN ))


mpirun -x OMP_NUM_THREADS \
mpirun -x OMP_NUM_THREADS -machinefile  $PBS_NODEFILE -np $PBS_NP \
    -machinefile  $PBS_NODEFILE \
     programname …
    -np $(wc -l < $PBS_NODEFILE) \
     …
</syntaxhighlight>
</syntaxhighlight>
The <code>-x</code> option is specific to OpenMPI; please consult the documentation to achieve the same behavior in other MPI implementations.
The <code>-x</code> option is specific to OpenMPI; please consult the documentation to achieve the same behavior in other MPI implementations.
Line 154: Line 186:
* The number of cores on the first node (running the job script) is the same as on the other nodes.
* The number of cores on the first node (running the job script) is the same as on the other nodes.
* All cores on a node will be used.
* All cores on a node will be used.
== Advanced: PBS_* Variables ==
The following environment variables are provided in a job environment, with their value being computed any job runtime.
<source lang="bash">
echo 'env | grep PBS_ | sort' | qsub -N inspect_env
# …
cat ~/inspect_env.o*
</source>
=== Particularly useful in job scripts ===
    '''PBS_O_WORKDIR'''=/home/username/foo/bar # directory qsub was running in (or its -w ''dir'' option)
    '''PBS_NP'''=1 # number of processors requested, '''automatically used for "mpirun -n …"'''
    '''PBS_NUM_NODES'''=1 # number of nodes, from resource "-l nodes=…"
    '''PBS_NUM_PPN'''=1 # ppn value, ditto
    '''PBS_NODEFILE'''=/var/spool/torque/aux/… # '''automatically used as "mpirun -machinefile …"''' option
    PBS_GPUFILE=/var/spool/torque/aux/… # file containing GPUs to access, present only with "-l nodes=…:gpus=…"
    PBS_MICFILE=/var/spool/torque/aux/… # (not applicable on Carbon)
    '''PBS_WALLTIME'''=3600 # requested (or default) walltime, in seconds
    '''PBS_ENVIRONMENT''' # useful for detection in scripts. Usually "PBS_BATCH", but "PBS_INTERACTIVE" under "qsub -I"
    '''PBS_JOBNAME'''=''jobname'' # from -N option, useful in job script for file names
=== Unix-like basics ===
    PBS_O_HOME=/home/username
    PBS_O_HOST=n123.carboncluster
    PBS_O_LANG=C # Unix locale(7), used for date formats, sorting, etc.
    PBS_O_LOGNAME=username
    PBS_O_MAIL=/var/spool/mail/username
    PBS_O_PATH=/home/username/bin:…
    PBS_O_SHELL=/bin/bash
=== PBS internals ===
    PBS_JOBCOOKIE=643…31D7
    PBS_JOBID=3141593.sched5.carboncluster
    PBS_MOMPORT=15003
    PBS_QUEUE=batch
    PBS_TASKNUM=1
    PBS_NODENUM=0 # usually always 0 (even on daughter nodes!)
    PBS_VNODENUM=0 # ditto.
    PBS_VERSION=TORQUE-4.2.10
    PBS_O_QUEUE=batch
    PBS_O_SERVER=sched5
    PBS_O_SUBMIT_FILTER=/usr/local/sbin/torque_submitfilter
<!--
$ strings /usr/sbin/pbs_mom | grep -E "^[A-Z]+_[A-Z_]{5,}$" | grep -Ev '^(A[UVW]|SIG|SUB|GLIB|CXX|MPI|WAIT|RADIX|NOT)' | sort | pr -w100 -Tv3
BATCH_ALLOC_COOKIE              MOAB_NODELIST                    PBS_NNODES
BATCH_PARTITION_ID              MOAB_PARTITION                  PBS_NODEFILE
BEOWULF_JOB_MAP                  MOAB_PROCCOUNT                  PBS_NODENUM
CLUSTER_ADDRS                    MOAB_TASKMAP                    PBS_NUM_NODES
CUDA_VISIBLE_DEVICES            OFFLOAD_DEVICES                  PBS_NUM_PPN
END_OF_LINE                      PBS_BATCH                        PBS_O_HOST
GET_TASKS                        PBS_BATCH_SERVICE_PORT          PBS_O_INITDIR
GPU_STATUS                      PBS_ENVIRONMENT                  PBS_O_ROOTDIR
JOIN_JOB_RADIX                  PBS_GPUFILE                      PBS_O_WORKDIR
KILL_JOB_RADIX                  PBS_INTERACTIVE                  PBS_QUEUE
MOAB_CLASS                      PBS_JOBCOOKIE                    PBS_RESOURCE_GRES
MOAB_GROUP                      PBS_JOBID                        PBS_RESOURCE_NODES
MOAB_JOBARRAYINDEX              PBS_JOBNAME                      PBS_SCHED_HINT
MOAB_JOBARRAYRANGE              PBS_MANAGER_SERVICE_PORT        PBS_TASKNUM
MOAB_JOBID                      PBS_MICFILE                      PBS_VERSION
MOAB_JOBNAME                    PBS_MOMPORT                      PBS_VNODENUM
MOAB_MACHINE                    PBS_MOM_SERVICE_PORT            PBS_WALLTIME
MOAB_NODECOUNT                  PBS_MSHOST
-->
[[Category:HPC]]
[[Category:HPC]]

Latest revision as of 17:41, August 30, 2024

Node Types

Hardware

Carbon has many major node generations, named genN for short, with N being an integer. In some generations, nodes differ further by the amount of memory.

Node Types

Node
names, types 		Node
generation 		Node
extra
properties 		Node
count 		Cores
per node
(max. ppn) 		Cores total,
by type 		Account
charge
rate 		CPU
model 		CPUs
per node 		CPU
nominal
clock
(GHz) 		Mem.
per node
(GB) 		Mem.
per core
(GB) 		GPU
model 		GPU
per node 		VRAM
per GPU
(GB) 		Disk
per node
(GB) 		Year
added 		Note
Login
login5…6 gen7a gpus=2 2 16 32 1.0 Xeon Silver 4125 2 2.50 192 12 Tesla V100 2 32 250 2019
Compute
n421…460 gen5 40 16 640 1.0 Xeon E5-2650 v4 2 2.10 128 8 250 2017
n461…476 gen6 16 16 256 1.0 Xeon Silver 4110 2 2.10 96 6 1000 2018
n477…512 gen6 36 16 576 1.0 Xeon Silver 4110 2 2.10 192 12 1000 2018
n513…534 gen7 gpus=2 22 32 704 1.5 Xeon Gold 6226R 2 2.90 192 6 Tesla V100S 2 32 250 2020
n541…580 gen8 20 64 2560 1.0 Xeon Gold 6430 2 2.10 1024 16 420 2024
Total 134 4736 48
  • Compute time is charged as the product of cores reserved × wallclock time × charge rate. The charge rate accommodates nominal differences in CPU speed.
  • gen7 nodes have two GPUs each; GPU usage is currently not "charged" (accounted for) separately.
  • Virtual memory usage on nodes may reach up to about 2 × the physical memory size. Your processes running under PBS may allocate that much vmem but cannot practically use it all for reasons of swap space size and bandwidth. If a node acitvely uses swap for more than a few minutes (which drastically slows down compute performance), the job will automatically be killed.

Major CPU flags

CPU capabilities grow with each node generation. Executables can be compiled to leverage specific CPU capabilities. Jobs using such executables must use the qsub option -l nodes=...:genX to be directed to nodes having that capability.

Major CPU capability flags by node generation. For details, see: CPUID instruction in Wikipedia, a StackExchange article, or /usr/src/kernels/*/arch/x86/include/asm/cpufeatures.h in kernel sources.
Flag name gen5 gen6 gen7 gen8
cat_l2 cdp_l2 cldemote gfni movdir64b movdiri pconfig sha_ni umip vaes vpclmulqdq – – – x
avx512_bitalg – – – x
avx512_vbmi2 – – – x
avx512_vpopcntdq – – – x
avx512ifma – – – x
avx512vbmi – – – x
avx512_vnni – – x x
mpx – x x –
avx512bw – x x x
avx512cd – x x x
avx512dq – x x x
avx512f – x x x
avx512vl – x x x
art clwb flush_l1d ibpb mba md_clear ospke pku ssbd stibp tsc_deadline_timer xgetbv1 xsavec – x x x
3dnowprefetch abm acpi aes aperfmperf apic arat arch_perfmon bmi1 bmi2 bts cat_l3 cdp_l3 cmov constant_tsc cqm cqm_llc cqm_mbm_local cqm_mbm_total cqm_occup_llc cx16 cx8 dca de ds_cpl dtes64 dtherm dts eagerfpu epb ept erms est f16c flexpriority fpu fsgsbase fxsr hle ht ida invpcid invpcid_single lahf_lm lm mca mce mmx monitor movbe msr mtrr nonstop_tsc nopl nx pae pat pbe pcid pclmulqdq pdcm pdpe1gb pebs pge pln pni popcnt pse pse36 pts rdrand rdseed rdt_a rdtscp rep_good rsb_ctxsw rtm sdbg sep smap smep smx ss sse sse2 sse4_1 ssse3 syscall tm tm2 tpr_shadow tsc tsc_adjust vme vmx vnmi vpid x2apic xsave xsaveopt xtopology xtpr x x x x
avx x x x x
avx2 x x x x
fma x x x x
adx x x x x
sse4_2 x x x x

Selecting node types for jobs

Jobs are directed automatically onto nodes of the same generation, with preference for gen2. Unless specifically requested, jobs will never mix generations. This will avoid disparate CPU speeds and MPI communication setup in a job. You can force jobs onto either node set in the job script after #PBS or on the qsub command line by suffixing the nodes= specifier with a property such as :gen6 or :gen8. For example, to run on 2 nodes with 8 cores each:


Recommendations

  • Avoid specifying a node generation unless you have a reason. This widens the pool of nodes suitable to run your job, thus decreasing wait times.
  • With ppn=16 your jobs may run on any node generation.

Advanced control of processeors-per-node (PPN)

Each Carbon node has nodes with different CPU core counts. A request of ppn=16 in the job submission can get routed onto a node with exactly 16 cores, or the job can get routed to a node with more cores, shareable by other jobs.

You can specify if your application is fine to run on shared nodes or needs nodes for exclusive use, such as for the following reasons:

  • your application is not parallelized,
  • your application has limited hardcoded parallelization, e.g. for 2 or 4 cores only,
  • your application runs multi-threaded but uses $PBS_NODEFILE to infer the number of processes to start,
  • your application runs busy service processes or service threads (e.g. NWChem),
  • your application saturates a resource, e.g. memory bandwidth (some large VASP calculations),
  • the node's memory is exhausted by fewer application processeses than there are cores available.

Depending on the reason, the node either may be or must not be used by other jobs. In the past, the only way to achieve exclusive but undersubscribed node access was to request ppn=8 and then to thin out a copy of the nodefile before passing it to the application. To eliminate the need to edit the nodefile, use the -l naccesspolicy=… flag to differentiate between resources requested from Moab from those passed to the application (in $PBS_NODEFILE).

Select an option from the following scenarios.

Shared vs. Exclusive Node Access

Permit other users and jobs
When a job requires only a few cores and a commensurate fraction of other resources, simply specify ppn as needed:
#PBS -l nodes=nnn:ppn=4
In this case, the remaining cores may be allocated to other jobs, which is the default policy:
#PBS -l naccesspolicy=SHARED
Permit only your own jobs
#PBS -l nodes=nnn:ppn=2
#PBS -l naccesspolicy=SINGLEUSER -n
Permit only one job per node, no sharing
When your job requires only a few cores but a disproportionate fraction of another resource on a node (such as most of its memory or a lot of I/O bandwidth), claim the entire node:
#PBS -l nodes=nnn:ppn=4
#PBS -l naccesspolicy=SINGLEJOB -n
PBS will reserve the entire node(s), but place each node name only ppn times in the $PBS_NODEFILE. This is also useful for MPI+OpenMP ("hybrid") programming, see below.
Permit only one of your jobs, and permit other user's jobs
#PBS -l nodes=nnn:ppn=4
#PBS -l naccesspolicy=UNIQUEUSER
The node is shared, but limited to one job for any given user.

Memory constraints

Jobs that do not use all cores on a node are subject to memory contraints, such that memory on each node is allocated in rough proportion to the number of cores requested. Specifically, most of Carbon's current nodes have 2 physical memory banks (sets of DIMM slots), and only one of these is made accessible to jobs that request ppn ≤ nproc/2, i.e., half the number of physical cores per node nproc.

To request access to all memory on a node for jobs with ppn < nproc use the qsub -n option, as shown in the solutions above.

Different PPN by node

When your first MPI process (the "master" process) requires more memory than your other "worker" processes, give several nodes specifications, separated by a "+" character (which is unusal and born of historical necessity):
#PBS -l nodes=1:ppn=1+2:ppn=4
#PBS -l naccesspolicy=SINGLEJOB -n
For clarity, the nodes specification in this example reads as follows:
nodes = ( 1:ppn=1 ) + ( 2:ppn=4 )
This will request 3 node exclusively, but the first node will occur only once in the $PBS_NODEFILE, e.g.
n011
n012
n012
n012
n012
n034
n034
n034
n034

In all of the preceding scenarios the following applies:

  • The $PBS_NODEFILE seen by the job script will always match ppn.
  • For accounting, the job will be billed by the number of cores blocked from use by other users, i.e., ncores=ppn for shared nodes, and ncores=8 otherwise.

Multithreading using OpenMP

When you wish to use multithreading, you must ensure that the total number of "busy" user threads and processes corresponds to the number of cores requested from PBS. Today, multithreading in applications and libraries is typically programmed using the OpenMP interface and the number of threads is controlled by the environment variable $OMP_NUM_THREADS. Select from the following scenarios.

Pure OpenMP, single entire node

#PBS -l nodes=1:ppn=8
#PBS -l naccesspolicy=SINGLEJOB -n

cd $PBS_O_WORKDIR
export OMP_NUM_THREADS=$PBS_NUM_PPN
...
programname …

Pure OpenMP, single node, possibly shared

Choose the number of cores n such that 1 ≤ n ≤ 8 (or 16): 

#PBS -l nodes=1:ppn=n
...
cd $PBS_O_WORKDIR
export OMP_NUM_THREADS=$PBS_NUM_PPN
...
programname …

Here, the default policy "SHARED" is in effect, and OMP_NUM_THREADS is set automatically by counting the number of times that the first node occurs in $PBS_NODEFILE. This will allow you to vary or override the nodes setting using "qsub -l nodes=…" without having to edit it twice in the job file.

OpenMP/MPI hybrid

Making efficient use of multithreading on multiple nodes which communicate over MPI is fairly involved and is subject to ongoing research. Since OMP_NUM_THREADS is set to 1 by default on MPI satellite nodes, you must export this variable after you altered it in the job file. 

#!/bin/bash
#PBS -l nodes=nnn:ppn=2
#PBS -l naccesspolicy=SINGLEJOB -n

# Calculate number of threads available per MPI process
cores_per_node=$( grep -c ^processor /proc/cpuinfo )
export OMP_NUM_THREADS=$(( cores_per_node / PBS_NUM_PPN ))

mpirun -x OMP_NUM_THREADS -machinefile  $PBS_NODEFILE -np $PBS_NP \
     programname …

The -x option is specific to OpenMPI; please consult the documentation to achieve the same behavior in other MPI implementations.

The last example will ensure:

  • you get allocated entire nodes (SINGLEJOB policy)
  • you do not oversubscribe cores (OMP_NUM_THREADS is calculated from ppn)
  • you only have one place to adjust (ppn), and can do so in the command line, or even post submission

It is assumed:

  • The number of cores on the first node (running the job script) is the same as on the other nodes.
  • All cores on a node will be used.

Advanced: PBS_* Variables

The following environment variables are provided in a job environment, with their value being computed any job runtime. 

echo 'env | grep PBS_ | sort' | qsub -N inspect_env
# …
cat ~/inspect_env.o*

Particularly useful in job scripts

   PBS_O_WORKDIR=/home/username/foo/bar		# directory qsub was running in (or its -w dir option)

   PBS_NP=1					# number of processors requested, automatically used for "mpirun -n …"
   PBS_NUM_NODES=1				# number of nodes, from resource "-l nodes=…"
   PBS_NUM_PPN=1				# ppn value, ditto

   PBS_NODEFILE=/var/spool/torque/aux/… 	# automatically used as "mpirun -machinefile …" option
   PBS_GPUFILE=/var/spool/torque/aux/…		# file containing GPUs to access, present only with "-l nodes=…:gpus=…"
   PBS_MICFILE=/var/spool/torque/aux/…		# (not applicable on Carbon)

   PBS_WALLTIME=3600				# requested (or default) walltime, in seconds
   PBS_ENVIRONMENT				# useful for detection in scripts. Usually "PBS_BATCH", but "PBS_INTERACTIVE" under "qsub -I"
   PBS_JOBNAME=jobname				# from -N option, useful in job script for file names

Unix-like basics

   PBS_O_HOME=/home/username
   PBS_O_HOST=n123.carboncluster
   PBS_O_LANG=C					# Unix locale(7), used for date formats, sorting, etc.
   PBS_O_LOGNAME=username
   PBS_O_MAIL=/var/spool/mail/username
   PBS_O_PATH=/home/username/bin:…
   PBS_O_SHELL=/bin/bash

PBS internals

   PBS_JOBCOOKIE=643…31D7
   PBS_JOBID=3141593.sched5.carboncluster
   PBS_MOMPORT=15003
   PBS_QUEUE=batch
   PBS_TASKNUM=1 
   PBS_NODENUM=0				# usually always 0 (even on daughter nodes!)
   PBS_VNODENUM=0				# ditto.
   PBS_VERSION=TORQUE-4.2.10
   PBS_O_QUEUE=batch
   PBS_O_SERVER=sched5
   PBS_O_SUBMIT_FILTER=/usr/local/sbin/torque_submitfilter
Retrieved from "https://wiki.anl.gov/wiki_cnm/index.php?title=HPC/Submitting_and_Managing_Jobs/Advanced_node_selection&oldid=7309"