HPC/Submitting and Managing Jobs

Environment configuration

Home dir

The users' home directories are hosted on lustre and are backed up nightly. The home directory can be reached, in standard Unix fashion using either of the two following symbols:

~
$HOME

lustre sandbox

For files that need to be shared among the nodes, and are possibly large and change often, use a "sandbox" directory. The environment variable

$SANDBOX

points to such a user-specific directory, which is shared by lustre, but not backed up. lustre is a parallel file system that allows concurrent and coherent file access at high data rates.

Applications

In the final configuration, we will use the environment-modules package to manage user applications. This will be similar to places like NERSC or PNNL. In early access mode, the CNM-specific user environment is configured automatically in /etc/profile.d/cnm.{sh,csh}.

For now the only applications are the Development tools.

Admin note: The master copy of these files resides in mgmt{01,02}:/opt/teamhpc/node-skel/etc/profile.d and is distributed by ~root/bin/skeldistrib.

Submitting jobs to Moab/Torque

 qsub [-A accountname] [options] jobfile

For details on options:

 man qsub
 qsub --help     # sorry, not much)

We currently have only the default queue configured.

More details at the Torque Wiki, in particular the full qsub documentation for all supported PBS options.

Querying jobs

Use the command qstat (from PBS) or showq (from Moab):

qstat

showq

regular output

qstat -a

showq -n

alternate format (showing names)

qstat -f

full information

checkjob [-v] jobnum

get extended jobs status information – useful to diagnose problems with "stuck" jobs.

Removing jobs

 qdel jobnumber

Example job file

• sample job file for Infiniband interconnect (recommended):

#!/bin/bash

##  Basics: Number of nodes, processors per node (ppn), and walltime (hhh:mm:ss)
#PBS -l nodes=5:ppn=8
#PBS -l walltime=0:10:00
#PBS -N job_name
#PBS -A account

## redirect stdout and stderr
#PBS -o job.out
#PBS -e job.err

## send mail at begin, end, abort, or never (b, e, a, n)
#PBS -m ea

# change directory
cd $PBS_O_WORKDIR

# set $NPROCS to the count of allocated cores
NPROCS=`wc -l < $PBS_NODEFILE`

# Verbose info, may be removed
cat << END_INFO
	master node:		`hostname`
	working directory:	`pwd`
	nodes file:		$PBS_NODEFILE
	number of processes:	$NPROCS
	process list:		`cat $PBS_NODEFILE`
END_INFO


# start MPI job over Infiniband transport
mpirun -machinefile $PBS_NODEFILE -np $NPROCS \
        programname

Update 2009-02-24

OpenIB is now the default, set in the environment variable OMPI_MCA_btl.

• To select ethernet transport (e.g. for embarrasingly parallel jobs), simply omit the -mca options:

mpirun -machinefile $PBS_NODEFILE -np $NPROCS \
	-mca btl self,tcp \
        programname

The account parameter

The parameter for option -A account is in most cases the CNM proposal, specified as follows:

cnm123

(3 digits) for proposals below 1000

cnm01234

(5 digits, 0-padded) for proposals from 1000 onwards.

user

for a limited personal startup allocation

staff

for discretionary access by staff.

Using OpenMP

For hybrid MPI/OpenMP operation under PBS (which is what happens when linking the MKL with OpenMP), two adjustments are necessary:

1. The environment variable OMP_NUM_THREADS needs to be set to the number of available cores per node, i.e., the ppn parameter. By default, this variable is set to 1 to select single-threading of OpenMP-compiled applications.
2. The machinefile needs to be thinned out in the job file, to have each node listed only once.

Example

#!/bin/bash
#PBS -l nodes=nnn:ppn=8
...
MACHINEFILE=$PBS_NODEFILE
...
if [ multithreaded ]            # insert specific condition
then
    sort -u $MACHINEFILE > machinefile
    MACHINEFILE=machinefile
    export OMP_NUM_THREADS=8
fi
...
NPROC=`wc -l < $MACHINEFILE`
...

Hybrid MPI+OpenMP is currently unsupported and may well be less efficient than compiling and running with MPI-only communication.

Policies

• Direct user access to nodes is only possible while a job is running for that user. This is governed by the torque-pam package.

• When attempting to schedule a job that used 5 nodes and 1 cpu per node (for a total of 5 cpus), Moab's default policy was to pack all of those jobs onto as few nodes as possible. Thus it was putting the job on 5 cores of the same node. This is usually not desired behavior, so I set a new parameter in /opt/moab/moab.cfg: "JOBNODEMATCHPOLICY EXACTNODE". Now jobs are allocated to nodes exactly as users request. [to be reviewed - stern]

-- Mail from W. Dinkel, Oct. 31