HPC/Submitting and Managing Jobs/Advanced node selection
Node Types
Hardware
Carbon has several major hardware node types, named genX for short, with X = 1, 2. …. Nodes are further sub-classified by their amount of installed memory.
Node Types
Node names, types |
Node generation |
Node extra properties |
Node count |
Cores per node (max. ppn )
|
Cores total, by type |
Account charge rate |
CPU model |
CPU per node |
CPU nom. clock (GHz) |
GPU model |
GPU per node |
VRAM per GPU (GB) |
Memory per node |
Memory per core (GB) |
Disk per node (GB) |
Year added |
Note |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Login | |||||||||||||||||
login5…6 | gen7a | gpus=2 | 2 | 16 | 32 | 3.0 | Xeon Silver 4125 | 2 | 2.50 | Tesla V100 | 2 | 32 | 192 | 12 | 250 | 2019 | |
Compute | |||||||||||||||||
n241…288 | gen2 | 48 | 8 | 384 | 1.0 | Xeon E5540 | 2 | 2.53 | 48 | 3 | 250 | 2009 | phasing out | ||||
n401…408 | gen4 | 8 | 16 | 128 | 2.0 | Xeon E5-2650 v2 | 2 | 2.60 | 128 | 8 | 250 | 2014 | |||||
n409…420 | gen4 | 12 | 16 | 192 | 2.0 | Xeon E5-2650 v2 | 2 | 2.60 | 64 | 4 | 250 | 2014 | |||||
n421…460 | gen5 | 40 | 16 | 640 | 2.0 | Xeon E5-2650 v4 | 2 | 2.10 | 128 | 8 | 250 | 2017 | |||||
n461…476 | gen6 | 16 | 16 | 256 | 2.0 | Xeon Silver 4110 | 2 | 2.10 | 96 | 6 | 1000 | 2018 | |||||
n477…512 | gen6 | 36 | 16 | 576 | 2.0 | Xeon Silver 4110 | 2 | 2.10 | 192 | 12 | 1000 | 2018 | |||||
n513…534 | gen7 | gpus=2 | 22 | 32 | 704 | 3.0 | Xeon Gold 6226R | 2 | 2.90 | Tesla V100S | 2 | 32 | 192 | 6 | 250 | 2020 | Installation in progress |
Retired | |||||||||||||||||
n001…144 | gen1 | 144 | 8 | 1152 | 0.4 | Xeon X5355 | 2 | 2.67 | 16 | 2 | 160 | 2007–2015 | retired (all) | ||||
n145…240,289…348 | gen2 | 156 | 8 | 1392 | 1.0 | Xeon E5540 | 2 | 2.53 | 24 | 3 | 250 | 2009-2019 | retired (≈70%) | ||||
n352…366 | gen3 | gpus=1 | 15 | 12 | 180 | 1.0 | Xeon E5645 | 2 | 2.40 | Tesla C2075 | 6 | 48 | 4 | 250 | 2012 | retired |
- Compute time is charged as the product of cores reserved × wallclock time × charge rate. The charge rate accommodates nominal differences in CPU speed. The reference speed (100%) is taken at a gen2 core.
- Compute time on gen4 nodes is charged at 200% of actual walltime, given their performance relative to gen2 nodes.
- gen7 nodes have two GPUs each; GPU usage is currently not "charged" (accounted for) separately.
- Virtual memory usage on nodes may reach up to about 2 × the physical memory size. Your processes running under PBS may allocate that much vmem but cannot practically use it all for reasons of swap space size and bandwidth. If a node acitvely uses swap for more than a few minutes (which drastically slows down compute performance), the job will automatically be killed.
CPU flags
The CPU capabilities grow with each node generation. Executables can be compiled to leverage specific CPU capabilities. Jobs using such executables must use the qsub option -l nodes=...:genX
to be directed to nodes having that capability.
Here is a list of capability flags by node generation, obtained from /proc/cpuinfo
.
For the meaning of the flags, see articles at StackExchange and Wikipedia.
Flag name | gen2 | gen4 | gen5 | gen6 | gen7 |
---|---|---|---|---|---|
avx512_vnni | – | – | – | – | x |
avx512vl, avx512f, avx512dq, avx512cd, avx512bw | – | – | – | x | x |
adx | – | – | x | x | x |
avx2 | – | – | x | x | x |
fma | – | – | x | x | x |
avx | – | x | x | x | x |
sse4_2 | x | x | x | x | x |
Selecting node types for jobs
Jobs are directed automatically onto nodes of the same generation, with preference for gen2. Unless specifically requested, jobs will never mix generations. This will avoid disparate CPU speeds and MPI communication setup in a job. You can force jobs onto either node set in the job script after #PBS
or on the qsub
command line by suffixing the nodes=
specifier with a property such as :gen1
or :gen2
. For example, to run on 2 nodes with 8 cores each:
qsub -l nodes=2:ppn=8:gen1 foo.job # not recommended for VASP
qsub -l nodes=2:ppn=8:gen2 foo.job
The following are (as of now) equivalent, since "bigmem" currently implies "gen2":
qsub -l nodes=2:ppn=8:gen2:bigmem foo.job
qsub -l nodes=2:ppn=8:bigmem foo.job
See also: http://www.clusterresources.com/torquedocs21/2.1jobsubmission.shtml#resources
Recommendations
- Avoid specifying a node generation unless you have a reason. This widens the pool of nodes suitable to run your job, thus decreasing wait times.
- To get you high memory per core, specify
:bigmem
(6 GB) or:hugemem
(8 GB, charged at 2 × walltime rate). - With
ppn=8
your jobs may run on any node generation. - With
ppn=4
your jobs have a fairly good chance of running in the "attic" of gen3 nodes which are often busy withppn=8
jobs.
PPN Tricks
Each Carbon node has nodes with 8, 12, and 16 cores.
A request of ppn=8
in the job submission can get routed onto a node with exactly 8 cores and thus use the entire node,
or the job can get routed to a node with additional cores which may be used by other jobs.
You can specify if your application is fine to run on shared nodes or needs nodes for exclusive use, such as for the following reasons:
- your application is not parallelized,
- your application has limited hardcoded parallelization, e.g. for 2 or 4 cores only,
- your application runs multi-threaded but uses
$PBS_NODEFILE
to infer the number of processes to start, - your application runs busy service processes or service threads (e.g. NWChem),
- your application saturates a resource, e.g. memory bandwidth (some large VASP calculations),
- the node's memory is exhausted by fewer application processeses than there are cores available.
Depending on the reason, the node either may be or must not be used by other jobs.
In the past, the only way to achieve exclusive but undersubscribed node access was to request ppn=8
and then to thin out a copy of the nodefile before passing it to the application.
To eliminate the need to edit the nodefile, use the -l naccesspolicy=…
flag to differentiate between resources requested from Moab from those passed to the application (in $PBS_NODEFILE).
Select an option from the following scenarios.
- Permit other users and jobs
- When a job requires only a few cores and a commensurate fraction of other resources, simply specify
ppn
as needed:
#PBS -l nodes=nnn:ppn=4
- In this case, the remaining cores may be allocated to other jobs, which is the default policy:
#PBS -l naccesspolicy=SHARED
- Permit only your own jobs
#PBS -l nodes=nnn:ppn=2 #PBS -l naccesspolicy=SINGLEUSER -n
- Permit only one job per node, no sharing
- When your job requires only a few cores but a disproportionate fraction of another resource on a node (such as most of its memory or a lot of I/O bandwidth), claim the entire node:
#PBS -l nodes=nnn:ppn=4 #PBS -l naccesspolicy=SINGLEJOB -n
- PBS will reserve the entire node(s), but place each node name only
ppn
times in the$PBS_NODEFILE
. This is also useful for MPI+OpenMP ("hybrid") programming, see below. - Permit only one of your jobs, and permit other user's jobs
#PBS -l nodes=nnn:ppn=4 #PBS -l naccesspolicy=UNIQUEUSER
- The node is shared, but limited to one job for any given user.
Memory constraints
Jobs that do not use all cores on a node are subject to memory contraints,
such that memory on each node is allocated in rough proportion to the number of cores requested.
Specifically, most of Carbon's current nodes have 2 physical memory banks (sets of DIMM slots),
and only one of these is made accessible to jobs that request ppn ≤ nproc/2
,
i.e., half the number of physical cores per node nproc
.
To request access to all memory on a node for jobs with ppn < nproc
use the qsub -n
option, as shown in the solutions above.
Different PPN by node
- When your first MPI process (the "master" process) requires more memory than your other "worker" processes, give several
nodes
specifications, separated by a"+"
character (which is unusal and born of historical necessity):
#PBS -l nodes=1:ppn=1+2:ppn=4 #PBS -l naccesspolicy=SINGLEJOB -n
- For clarity, the
nodes
specification in this example reads as follows:
nodes = ( 1:ppn=1 ) + ( 2:ppn=4 )
- This will request 3 node exclusively, but the first node will occur only once in the
$PBS_NODEFILE
, e.g.
n011 n012 n012 n012 n012 n034 n034 n034 n034
In all of the preceding scenarios the following applies:
- The
$PBS_NODEFILE
seen by the job script will always matchppn
. - For accounting, the job will be billed by the number of cores blocked from use by other users, i.e.,
ncores=ppn
for shared nodes, andncores=8
otherwise.
Multithreading using OpenMP
When you wish to use multithreading, you must ensure that the total number of "busy" user threads and processes corresponds to the number of cores requested from PBS. Today, multithreading in applications and libraries is typically programmed using the OpenMP interface and the number of threads is controlled by the environment variable $OMP_NUM_THREADS
.
Select from the following scenarios.
Pure OpenMP, single entire node
#PBS -l nodes=1:ppn=8
#PBS -l naccesspolicy=SINGLEJOB -n
cd $PBS_O_WORKDIR
export OMP_NUM_THREADS=$PBS_NUM_PPN
...
programname …
Choose the number of cores n
such that 1 ≤ n ≤ 8 (or 16)
:
#PBS -l nodes=1:ppn=n
...
cd $PBS_O_WORKDIR
export OMP_NUM_THREADS=$PBS_NUM_PPN
...
programname …
Here, the default policy "SHARED" is in effect, and OMP_NUM_THREADS is set automatically by counting the number of times that the first node occurs in $PBS_NODEFILE
. This will allow you to vary or override the nodes setting using "qsub -l nodes=…" without having to edit it twice in the job file.
OpenMP/MPI hybrid
Making efficient use of multithreading on multiple nodes which communicate over MPI is fairly involved and is subject to ongoing research. Since OMP_NUM_THREADS
is set to 1 by default on MPI satellite nodes, you must export this variable after you altered it in the job file.
#!/bin/bash
#PBS -l nodes=nnn:ppn=2
#PBS -l naccesspolicy=SINGLEJOB -n
# Calculate number of threads available per MPI process
cores_per_node=$( grep -c ^processor /proc/cpuinfo )
export OMP_NUM_THREADS=$(( cores_per_node / PBS_NUM_PPN ))
mpirun -x OMP_NUM_THREADS -machinefile $PBS_NODEFILE -np $PBS_NP \
programname …
The -x
option is specific to OpenMPI; please consult the documentation to achieve the same behavior in other MPI implementations.
The last example will ensure:
- you get allocated entire nodes (SINGLEJOB policy)
- you do not oversubscribe cores (OMP_NUM_THREADS is calculated from ppn)
- you only have one place to adjust (ppn), and can do so in the command line, or even post submission
It is assumed:
- The number of cores on the first node (running the job script) is the same as on the other nodes.
- All cores on a node will be used.
Advanced: PBS_* Variables
The following environment variables are provided in a job environment, with their value being computed, from torque-3.0.5/src/resmom/start_exec.c :
static char *variables_else[] = /* variables to add, value computed */
{
"HOME",
"LOGNAME",
"PBS_JOBNAME",
"PBS_JOBID",
"PBS_QUEUE",
"SHELL",
"USER",
"PBS_JOBCOOKIE",
"PBS_NODENUM",
"PBS_TASKNUM",
"PBS_MOMPORT",
"PBS_NODEFILE",
"PBS_NNODES", /* number of nodes specified by size */
"TMPDIR",
"PBS_VERSION",
"PBS_NUM_NODES", /* number of nodes specified by nodes string */
"PBS_NUM_PPN", /* ppn value specified by nodes string */
"PBS_GPUFILE", /* file containing which GPUs to access */
"PBS_NP", /* number of processors requested */
"PBS_WALLTIME", /* requested or default walltime */
NULL
};