HPC/Submitting and Managing Jobs

Directories and Environment

First read: directory configuration.

Applications

We use the environment-modules package to manage user applications. This is similar to places like NERSC or PNNL. The basic CNM-specific user environment is configured automatically in /etc/profile.d/cnm.{sh,csh}.

For now the only applications are the Development tools.

Admin note: The master copy of these files resides in mgmt{01,02}:/opt/teamhpc/node-skel/etc/profile.d and is distributed by ~root/bin/skeldistrib.

Submitting jobs to Moab/Torque

 qsub [-A accountname] [options] jobfile

For details on options:

 man qsub
 qsub --help

More details are at the Torque Manual, in particular the qsub man page.

The single main queue is batch and need not be specified. All job routing decisions are handled by the scheduler. In particular, short jobs are accommodated by a daily reserved node and by backfill scheduling, i.e. "waving forward" while a big job waits for full resources to become available.

Debug queue

For testing job processing and your job environment, use qsub -q debug or #PBS -q debug. The queue accepts jobs under the following conditions

nodes <= 2
ppn <= 4
walltime <= 1:00:00

Querying jobs

Use the command qstat (from PBS) or showq (from Moab):

qstat [-u $USER]

showq [-u $USER]

regular output

qstat -a

showq -n

alternate format (showing names)

qstat -f [jobnum]

full information

checkjob [-v] jobnum

get extended jobs status information – useful to diagnose problems with "stuck" jobs.

Removing jobs

 qdel jobnumber

Example job file

• sample job file for Infiniband interconnect (recommended):

#!/bin/bash

##  Basics: Number of nodes, processors per node (ppn), and walltime (hhh:mm:ss)
#PBS -l nodes=5:ppn=8
#PBS -l walltime=0:10:00
#PBS -N job_name
#PBS -A account

## File names for stdout and stderr.  If not set here, the defaults
## are <JOBNAME>.o<JOBNUM> and <JOBNAME>.e<JOBNUM>
#PBS -o job.out
#PBS -e job.err

## send mail at begin, end, abort, or never (b, e, a, n)
#PBS -m ea

# change into the directory where qsub will be executed
cd $PBS_O_WORKDIR

# count allocated cores
NPROCS=`wc -l < $PBS_NODEFILE`

# start MPI job over default interconnect
mpirun -machinefile $PBS_NODEFILE -np $NPROCS \
        programname

• If your program reads from files or takes options and/or arguments, use and adjust one of the following forms

mpirun -machinefile $PBS_NODEFILE -np $NPROCS \
       programname  < run.in

mpirun -machinefile $PBS_NODEFILE -np $NPROCS \
       programname  -options arguments < run.in

mpirun -machinefile $PBS_NODEFILE -np $NPROCS \
       programname < run.in > run.out 2> run.err

In this form, anything after programname is optional. If you use specific redirections for stdout or stderr as shown (>, 2>), the job-global files job.out, job.err declared earlier will remain empty or only contain output from your shell startup files (which should really be silent), and the rest of your job script.

• Infiniband (OpenIB) is the default (and fast) interconnect mechanism for MPI jobs. This is configured through the environment variable $OMPI_MCA_btl.
• To select ethernet transport (e.g. for embarrasingly parallel jobs), specify an -mca option:

mpirun -machinefile $PBS_NODEFILE -np $NPROCS \
	-mca btl self,tcp \
        programname

The account parameter

The parameter for option -A account is in most cases the CNM proposal, specified as follows:

cnm123

(3 digits) for proposals below 1000

cnm01234

(5 digits, 0-padded) for proposals from 1000 onwards.

user

for a limited personal startup allocation

staff

for discretionary access by staff.

You can check your account balance in hours as follows:

mybalance -h
gbalance -u $USER -h

Using OpenMP

For hybrid MPI/OpenMP operation under PBS (which is what happens when linking the MKL with OpenMP), two adjustments are necessary:

1. The environment variable OMP_NUM_THREADS needs to be set to the number of available cores per node, i.e., the ppn parameter. By default, this variable is set to 1 to select single-threading of OpenMP-compiled applications.
2. The machinefile needs to be thinned out in the job file, to have each node listed only once.

Example

#!/bin/bash
#PBS -l nodes=nnn:ppn=8
...
MACHINEFILE=$PBS_NODEFILE
...
if [ multithreaded ]            # insert specific condition
then
    sort -u $MACHINEFILE > machinefile
    MACHINEFILE=machinefile
    export OMP_NUM_THREADS=8
fi
...
NPROC=`wc -l < $MACHINEFILE`
...

PPN Tricks

You may need to request fewer cores than a whole node physically offers, for example, due to the following reasons:

• the application is not parallelized
• the application has limited hardcoded parallelization, e.g. for 2 or 4 cores only (Carbon nodes have 8 cores)
• the application runs multi-threaded but uses $PBS_NODEFILE to infer processes to start
• the application runs busy service processes or service threads (e.g. NWChem)
• the application saturates a resource, e.g. memory bandwidth (some large VASP calculations)
• the node's memory is not sufficient to run processes on all cores.

Shared vs. Exclusive Node Access

Shared node access

This is the default. A job requests fewer cores than the node has; free cores can be allocated to other jobs. Simply specify ppn less than 8:

#PBS -l nodes=__:ppn=4

Exclusive node access

This may be needed when a job requires nearly the whole memory of a node, or nearly all of its I/O bandwidth.

 #PBS -l nodes=__:ppn=4 -l naccesspolicy=SINGLEJOB

Multithreading

When using multithreading (typically through OpenMP features), you must ensure that the number of "busy" user threads and processes corresponds to the number of cores requested from PBS. $PBS_NODEFILE to infer processes to start

OpenMP

#PBS -l nodes=__:ppn=1 -l naccesspolicy=SINGLEJOB
...
cd $PBS_O_WORKDIR
export OMP_NUM_THREADS=8
...

OpenMP + MPI Hybrid

This is still a very new field and subject to ongoing research. Usage is the same as the preceding case (OpenMP only), plus you must export OMP_NUM_THREADS to all MPI satellite nodes:

...
mpiexec -x OMP_NUM_THREADS …    # OpenMPI-specific

For either case the following holds:

• The $PBS_NODEFILE seen by the job script will always match ppn.
• Forr accounting, the job will be billed by the number of cores blocked from use by other users, i.e., the actual ppn for shared nodes, and ppn=8 for naccesspolicy=SINGLEJOB.

Policies

• Direct user access to nodes is only possible while a job is running for that user. This is governed by the torque-pam package.