HPC/Submitting and Managing Jobs: Difference between revisions

From CNM Wiki
< HPC
Jump to navigation Jump to search
m (PPN Tricks)
Line 106: Line 106:
  mybalance -h
  mybalance -h
  gbalance -u $USER -h
  gbalance -u $USER -h
<strike>
=== Using OpenMP ===
For [http://www.google.com/search?hl=en&rls=en-us&q=MPI+OpenMP hybrid MPI/OpenMP] operation under PBS (which is what happens when linking the MKL with OpenMP), two adjustments are necessary:
# The environment variable <code> OMP_NUM_THREADS</code> needs to be set to the number of available cores per node, i.e., the ''ppn'' parameter.  By default, this variable is set to 1 to select single-threading of OpenMP-compiled applications.
# The machinefile needs to be thinned out in the job file, to have each node listed only once.
==== Example ====
#!/bin/bash
#PBS -l nodes=''nnn'':ppn=8
...
MACHINEFILE=$PBS_NODEFILE
...
if [ ''multithreaded'' ]            # insert specific condition
then
    sort -u $MACHINEFILE > machinefile
    MACHINEFILE=machinefile
    export OMP_NUM_THREADS=8
fi
...
NPROC=`wc -l &lt; $MACHINEFILE`
...
</strike>


== PPN Tricks ==
== PPN Tricks ==
Line 142: Line 119:
; Shared node access:
; Shared node access:
: This is the default.  A job requests fewer cores than the node has; free cores can be allocated to other jobs.  Simply specify <code>ppn</code> less than 8:
: This is the default.  A job requests fewer cores than the node has; free cores can be allocated to other jobs.  Simply specify <code>ppn</code> less than 8:
  #PBS -l nodes=__:ppn=4
  #PBS -l nodes=''nnn'':ppn=4
#PBS -l '''naccesspolicy=SHARED'''
; User-specific node access:
: To share a node only among your own jobs, specify a ''SINGLEUSER'' policy:
#PBS -l nodes=''nnn'':ppn=4
#PBS -l '''naccesspolicy=SINGLEUSER'''
; Exclusive node access:
; Exclusive node access:
: This may be needed when a job requires nearly the whole memory of a node, or nearly all of its I/O bandwidth.
: When your job requires fre cores but ''all'' of a particular other resource on a node (such as most of its memory or a lot of I/O bandwidth), claim the entire node:
  #PBS -l nodes=__:ppn=4 '''-l naccesspolicy=SINGLEJOB'''
#PBS -l nodes=''nnn'':ppn=4
#PBS '''-l naccesspolicy=SINGLEJOB'''


=== Different PPN by node ===
Often, an MPI master process requires more memory than worker processes. Several <code>nodes</code> specifications can be given, separated by <code>"+"</code>:
#PBS -l nodes=1:ppn=1+3:ppn=8
#PBS -l naccesspolicy=SINGLEJOB
Requests 4 exclusive nodes, but the first node will occur only once in the <code>$PBS_NODEFILE</code>.
=== Multithreading ===
=== Multithreading ===
When using multithreading (typically through OpenMP features), you must ensure that the number of "busy" user threads and processes corresponds to the number of cores requested from PBS.
When using multithreading (typically through OpenMP features), you must ensure that the number of "busy" user threads and processes corresponds to the number of cores requested from PBS.
Line 153: Line 141:


; OpenMP:
; OpenMP:
  #PBS -l nodes=__:ppn=1 -l naccesspolicy=SINGLEJOB'''
  #PBS -l nodes=''nnn'':ppn=1 -l naccesspolicy=SINGLEJOB'''
  ...
  ...
  cd $PBS_O_WORKDIR
  cd $PBS_O_WORKDIR
Line 166: Line 154:
For either case the following holds:
For either case the following holds:
* The <code>$PBS_NODEFILE</code> seen by the job script will always match <code>ppn</code>.
* The <code>$PBS_NODEFILE</code> seen by the job script will always match <code>ppn</code>.
* Forr accounting, the job will be ''billed'' by the number of cores blocked from use by other users, i.e., the actual ppn for shared nodes, and ppn=8 for naccesspolicy=SINGLEJOB.
* For accounting, the job will be ''billed'' by the number of cores blocked from use by other users, i.e., the actual ppn for shared nodes, and ppn=8 for naccesspolicy=SINGLEJOB.
 
<!--
For [http://www.google.com/search?hl=en&rls=en-us&q=MPI+OpenMP hybrid MPI/OpenMP] operation under PBS (which is what happens when linking the MKL with OpenMP), two adjustments are necessary:
# The environment variable <code> OMP_NUM_THREADS</code> needs to be set to the number of available cores per node, i.e., the ''ppn'' parameter.  By default, this variable is set to 1 to select single-threading of OpenMP-compiled applications.
# The machinefile needs to be thinned out in the job file, to have each node listed only once.
 
#!/bin/bash
#PBS -l nodes=''nnn'':ppn=8
...
MACHINEFILE=$PBS_NODEFILE
...
if [ ''multithreaded'' ]            # insert specific condition
then
    sort -u $MACHINEFILE > machinefile
    MACHINEFILE=machinefile
    export OMP_NUM_THREADS=8
fi
...
NPROC=`wc -l &lt; $MACHINEFILE`
...
-->


=== Policies ===
=== Policies ===

Revision as of 01:37, November 6, 2010

Directories and Environment

First read: directory configuration.

Applications

We use the environment-modules package to manage user applications. This is similar to places like NERSC or PNNL. The basic CNM-specific user environment is configured automatically in /etc/profile.d/cnm.{sh,csh}.

For now the only applications are the Development tools.

Admin note: The master copy of these files resides in mgmt{01,02}:/opt/teamhpc/node-skel/etc/profile.d and is distributed by ~root/bin/skeldistrib.

Submitting jobs to Moab/Torque

 qsub [-A accountname] [options] jobfile

For details on options:

 man qsub
 qsub --help

More details are at the Torque Manual, in particular the qsub man page.

The single main queue is batch and need not be specified. All job routing decisions are handled by the scheduler. In particular, short jobs are accommodated by a daily reserved node and by backfill scheduling, i.e. "waving forward" while a big job waits for full resources to become available.

Debug queue

For testing job processing and your job environment, use qsub -q debug or #PBS -q debug. The queue accepts jobs under the following conditions

nodes <= 2
ppn <= 4
walltime <= 1:00:00

Querying jobs

Use the command qstat (from PBS) or showq (from Moab):

qstat [-u $USER]
showq [-u $USER]
regular output
qstat -a
showq -n
alternate format (showing names)
qstat -f [jobnum]
full information
checkjob [-v] jobnum
get extended jobs status information – useful to diagnose problems with "stuck" jobs.

Removing jobs

 qdel jobnumber

Example job file

  • sample job file for Infiniband interconnect (recommended):
#!/bin/bash

##  Basics: Number of nodes, processors per node (ppn), and walltime (hhh:mm:ss)
#PBS -l nodes=5:ppn=8
#PBS -l walltime=0:10:00
#PBS -N job_name
#PBS -A account

## File names for stdout and stderr.  If not set here, the defaults
## are <JOBNAME>.o<JOBNUM> and <JOBNAME>.e<JOBNUM>
#PBS -o job.out
#PBS -e job.err

## send mail at begin, end, abort, or never (b, e, a, n)
#PBS -m ea

# change into the directory where qsub will be executed
cd $PBS_O_WORKDIR

# count allocated cores
NPROCS=`wc -l < $PBS_NODEFILE`

# start MPI job over default interconnect
mpirun -machinefile $PBS_NODEFILE -np $NPROCS \
        programname
  • If your program reads from files or takes options and/or arguments, use and adjust one of the following forms
mpirun -machinefile $PBS_NODEFILE -np $NPROCS \
       programname  < run.in
mpirun -machinefile $PBS_NODEFILE -np $NPROCS \
       programname  -options arguments < run.in
mpirun -machinefile $PBS_NODEFILE -np $NPROCS \
       programname < run.in > run.out 2> run.err
In this form, anything after programname is optional. If you use specific redirections for stdout or stderr as shown (>, 2>), the job-global files job.out, job.err declared earlier will remain empty or only contain output from your shell startup files (which should really be silent), and the rest of your job script.
  • Infiniband (OpenIB) is the default (and fast) interconnect mechanism for MPI jobs. This is configured through the environment variable $OMPI_MCA_btl.
  • To select ethernet transport (e.g. for embarrasingly parallel jobs), specify an -mca option:
mpirun -machinefile $PBS_NODEFILE -np $NPROCS \
	-mca btl self,tcp \
        programname

The account parameter

The parameter for option -A account is in most cases the CNM proposal, specified as follows:

cnm123
(3 digits) for proposals below 1000
cnm01234
(5 digits, 0-padded) for proposals from 1000 onwards.
user
for a limited personal startup allocation
staff
for discretionary access by staff.

You can check your account balance in hours as follows:

mybalance -h
gbalance -u $USER -h

PPN Tricks

You may need to request fewer cores than a whole node physically offers, for example, due to the following reasons:

  • the application is not parallelized
  • the application has limited hardcoded parallelization, e.g. for 2 or 4 cores only (Carbon nodes have 8 cores)
  • the application runs multi-threaded but uses $PBS_NODEFILE to infer processes to start
  • the application runs busy service processes or service threads (e.g. NWChem)
  • the application saturates a resource, e.g. memory bandwidth (some large VASP calculations)
  • the node's memory is not sufficient to run processes on all cores.

Shared vs. Exclusive Node Access

Shared node access
This is the default. A job requests fewer cores than the node has; free cores can be allocated to other jobs. Simply specify ppn less than 8:
#PBS -l nodes=nnn:ppn=4
#PBS -l naccesspolicy=SHARED
User-specific node access
To share a node only among your own jobs, specify a SINGLEUSER policy:
#PBS -l nodes=nnn:ppn=4
#PBS -l naccesspolicy=SINGLEUSER
Exclusive node access
When your job requires fre cores but all of a particular other resource on a node (such as most of its memory or a lot of I/O bandwidth), claim the entire node:
#PBS -l nodes=nnn:ppn=4
#PBS -l naccesspolicy=SINGLEJOB

Different PPN by node

Often, an MPI master process requires more memory than worker processes. Several nodes specifications can be given, separated by "+":

#PBS -l nodes=1:ppn=1+3:ppn=8
#PBS -l naccesspolicy=SINGLEJOB

Requests 4 exclusive nodes, but the first node will occur only once in the $PBS_NODEFILE.

Multithreading

When using multithreading (typically through OpenMP features), you must ensure that the number of "busy" user threads and processes corresponds to the number of cores requested from PBS. $PBS_NODEFILE to infer processes to start

OpenMP
#PBS -l nodes=nnn:ppn=1 -l naccesspolicy=SINGLEJOB
...
cd $PBS_O_WORKDIR
export OMP_NUM_THREADS=8
...
OpenMP + MPI Hybrid
This is still a very new field and subject to ongoing research. Usage is the same as the preceding case (OpenMP only), plus you must export OMP_NUM_THREADS to all MPI satellite nodes:
...
mpiexec -x OMP_NUM_THREADS …    # OpenMPI-specific

For either case the following holds:

  • The $PBS_NODEFILE seen by the job script will always match ppn.
  • For accounting, the job will be billed by the number of cores blocked from use by other users, i.e., the actual ppn for shared nodes, and ppn=8 for naccesspolicy=SINGLEJOB.


Policies

  • Direct user access to nodes is only possible while a job is running for that user. This is governed by the torque-pam package.