HPC/Applications/g09
Introduction
Gaussian is an electronic structure program used by chemists, chemical engineers, biochemists, physicists and others for research in established and emerging areas of chemical interest.
Starting from the basic laws of quantum mechanics, Gaussian predicts the energies, molecular structures, and vibrational frequencies of molecular systems, along with numerous molecular properties derived from these basic computation types. It can be used to study molecules and reactions under a wide range of conditions, including both stable species and compounds which are difficult or impossible to observe experimentally such as short-lived intermediates and transition structures. This article introduces several of its new and enhanced features.
Version note
The g09 versions on Carbon are 64-bit versions with full support for shared memory and Linda parallelization.
$ module -l avail g09 - Package -----------------------------+- Versions -+- Last mod. ------ /opt/soft/modulefiles: g09/A.02.x86_64-1 2010/04/29 21:07:18 g09/A.02.x86_64-2 2011/08/05 19:56:04 g09/C.01.x86_64-1 2012/01/16 20:05:38 g09/D.01.x86_64-1 2013/08/02 19:19:22
The last version is the default at the moment for
module load g09
Status
Our license agreement with Gaussian, Inc. restricts the use of Gaussian applications to Argonne employees.
Job script
- Inspect the job script template (after having loaded the g09 module) at:
$G09_HOME/g09.job
- This script shows how to use a preprocessing script
g09preprocess
Usage
Copy the template script and use it in one of two ways:
- edit for each job as needed, or
- adapt the script for basic needs of several jobs, and individualize it by the PBS job name.
The job name can be up to 15 characters long and should not contain unusual characters. Set it, along with your project name and the walltime limit on the qsub command line:
qsub -N foo [-l walltime=hhh:mm:ss] g09.job
Operation
Upon job execution, the script reads the input file, and the g09preprocess script interprets and modifies your input for use under PBS on Carbon, as follows:
- Link0-commands
%NProcShared=ppnand%LindaWorkers=n123,n124,n125,...are inserted automatically. - Checkpoint-files named in
%chkdirectives are identified and copied into the compute node's job-specifc$TMPDIR. The%chkspecification will be changed to include$TMPDIR. Note that this will be echoed in the g03 output as follows:
%chk=/tmp/12345.sched1.carboncluster/test0420.chk
- Chk-file processing will be performed only for files within the current directory; to skip processing, specify a relative or absolute path, such as
%chk=./name.chk
- At the end of the job, chk-files in
$TMPDIRare moved to$PBS_O_WORKDIR.
Notes on Linda
G09 is parallelized in two complementary ways:
- on a single node across its processors using shared memory, and
- across different nodes via a subset of the Linda parallelization language – http://gaussian.com/g_tech/g_ur/m_linda.htm
Incomplete node use
g09/Linda may decide not to use the entirety of the nodes made available to it by the g09preprocess stage in the PBS job file.
This section describes how to determine when this is the case and what to do about it.
Consider the following input:
%mem=20mw #p MP2/6-311G(2df,p) force symm=loose MaxDisk=250000000 test …
This being run in the job file through g09preprocess:
#PBS -l nodes=2:ppn=8 … g09preprocess test0420.com | g09 > test0420.out
which produces the input that g09 will see, inserting the node names allocated by PBS:
%NProcShared=8 %LindaWorkers=n994,n992 %mem=20mw #p MP2/6-311G(2df,p) force symm=loose MaxDisk=250000000 test …
Gaussian will echo its input line by line in its log file:
Entering Gaussian System, Link 0=g09
Initial command:
/opt/soft/g09-D.01.x86_64-1/g09/l1.exe "/tmp/341710.sched1.carboncluster/Gau-32350.inp" -scrdir="/tmp/341710.sched1.carboncluster/"
Entering Link 1 = /opt/soft/g09-D.01.x86_64-1/g09/l1.exe PID= 32351.
Copyright (c) 1988,1990,1992,1993,1995,1998,2003,2009,2013,
Gaussian, Inc. All Rights Reserved.
…
Cite this work as:
Gaussian 09, Revision D.01,
…
******************************************
Gaussian 09: EM64L-G09RevD.01 24-Apr-2013
1-Aug-2013
******************************************
%NProcShared=8
Will use up to 8 processors via shared memory.
%LindaWorkers=n994,n992
%mem=20mw
SetLPE: input flags="-v -opt "Tsnet.Node.lindarsharg: ssh" "
SetLPE: new flags="-v -opt "Tsnet.Node.lindarsharg: ssh" -nodelist 'n994.carboncluster n992.carboncluster'"
Will use up to 2 processors via Linda.
--------------------------------------------------------------------
#p MP2/6-311G force …
--------------------------------------------------------------------
- The last line will be specific to your Gaussian route section. So far, so good.
- I highly recommend to use the
#p("profuse") flag to better track progress through Gaussian's link stages. - The term processors is used inconsistently here. For shared memory, it means cores, but for Linda it means nodes.
- You can track the use of Linda by:
grep exel file.out
(Enter /opt/soft/g09-D.01.x86_64-1/g09/linda-exe/l302.exel) (Enter /opt/soft/g09-D.01.x86_64-1/g09/linda-exe/l401.exel) …
- g09 may decide to not use Linda after all. A note like this will appear in the log file:
PrsmSu: requested number of processors reduced to: 6 ShMem 1 Linda.
- In this case, decide wether it would be better to resubmit the job than to leave nodes allocated by PBS idle. It is possible that g09's decision is different for different Linda links. Only the longest-running link would be relevant for you to evaluate resubmission.
Error messages
When running jobs on more than one node (i.e., using Linda), you may see messages like these in a job's error stream:
eval server 0 on n019.carboncluster has dropped it's connection. eval server 0 on n019.carboncluster has dropped it's connection. subprocess pid = 11505 has exited. status = 0x0000, id = 0, state = 17. command was /opt/soft/g09-D.01.x86_64-1/g09/linda8.2/opteron … died after signing in successfully eval server 0 on n019.carboncluster has dropped it's connection. …
- These messages can be ignored, according to http://umbc.rnet.missouri.edu/resources/How2RunGAUSSIAN.html :
These messages are not indicative of a problem. They indicate that the Linda work in a Gaussian link is finished, and that Gaussian is continuing with a new link.
- The environment variable GAUSS_FLAGS (as set in the g09 modulefile) is now configured to include the flag
-v, to better trace Linda operations on worker nodes. Additional lines like the following will show up in a job's error stream:
ntsnet: using executable file /opt/soft/g09-D.01.x86_64-1/g09/linda-exe/l302.exel ntsnet: trying to schedule 1 worker ntsnet: scheduled a total of 1 worker ntsnet: starting master process on n994.carboncluster ntsnet: starting 1 worker on n992.carboncluster
- This example shows how one link within a job of
#PBS -l nodes=2:ppn=8gets executed under Linda.