HPC/Directories

From CNM Wiki
< HPC
Revision as of 19:57, November 9, 2009 by Stern (talk | contribs) (→‎Use in job files: added)
Jump to navigation Jump to search

Overview

Here is a summary of key directories related to Carbon, and environment variables used to access them:

Environment variable Typical value Shared across nodes Notes
$HOME or ~ (tilde) /home/joe yes home sweet home
$SANDBOX /sandbox/joe yes extra storage, not backed up
$TMPDIR /tmp/pbs_mom/12345.mds01.... no job-specfic scratch
$PBS_O_WORKDIR (yes) the directory qsub was run in; typically used with cd $PBS_O_WORKDIR as first line in a job script

Details by function

Home directory

$HOME
~

The users' home directories are kept on a Lustre* file system and are backed up nightly. The home directory can be reached in standard Unix fashion using either the environment variable or the tilde sign in most shells (but generally not application programs, especially not those written in Fortran).

Sandbox - global scratch and overflow

$SANDBOX

For files that need to be shared among the nodes, and are possibly large and change often, use a "sandbox" directory. The environment variable points to a user-specific directory which is shared by Lustre*, but not backed up.


Local scratch space

$TMPDIR

This variable and the directory it refers to is provided by the queueing system for all processes that execute a job. The directory:

  • resides on local disk on each node
  • typically provides about 100 GB of space
  • is the same for [MPI] processes on the SAME node (as many as given in "ppn=…")
  • (i.e., PBS job-specific, not Unix PID-specific)
  • is not shared across nodes
  • will be wiped upon job exit

Use in job files

You can use $TMPDIR in one or more of the following ways:

  • direct your application to store its temporary files there, which is typically done by command line switches or an environment variable such as:
export FOO_SCRATCH=$TMPDIR
  • actually run your application there:
cd $TMPDIR
In this case, make sure you either copy your input files there or you specify full paths to $HOME or $PPBS_O_WORKDIR.
  • copy files back upon job termination:
#PBS -W stageout=$TMPDIR/foo.ext@localhost:$PBS_O_WORKDIR
#PBS -W stageout=$TMPDIR/*.bar@localhost:$PBS_O_WORKDIR
You may specify several of these lines and use wildcards in to specify source files on the compute nodes. In contrast to an explicit "cp" command in the job script, this copy operation will be executed even if a job overruns its walltime. See the qsub manual for further information.



(*) Lustre is a parallel file system that allows concurrent and coherent file access at high data rates.