HPC/Applications/gaussian

Introduction

Gaussian is an electronic structure program used by chemists, chemical engineers, biochemists, physicists and others for research in established and emerging areas of chemical interest.

Starting from the basic laws of quantum mechanics, Gaussian predicts the energies, molecular structures, and vibrational frequencies of molecular systems, along with numerous molecular properties derived from these basic computation types. It can be used to study molecules and reactions under a wide range of conditions, including both stable species and compounds which are difficult or impossible to observe experimentally such as short-lived intermediates and transition structures. This article introduces several of its new and enhanced features.

Version note

The Gaussian versions on Carbon are 64-bit versions with full support for shared memory and Linda parallelization.

module -l avail gaussian

gaussian/09/09.D.01.x86_64-3           2015/12/02 17:25:23
gaussian/16/16-A.03-1                  2017/10/16 12:42:41
gaussian/16/16-A.03-2                  2017/10/16 12:35:29
gaussian/16/16-B.01-1                  2019/05/28 16:56:37
gaussian/16/16-C.01-1                  2019/08/20 10:03:31

The last version is the default at present, obtained by the (recommended) version-free module command:

module load gaussian

License status

Our license agreement with Gaussian, Inc. restricts the use of Gaussian applications to Argonne employees.

Job script

• Inspect the job script template (after having loaded the module) as follows:

$GAUSSIAN_HOME/g16.job

Usage

Copy the template script and use it in one of two ways:

• edit for each job as needed, or
• adapt the script for basic needs of several jobs, and individualize it by the PBS job name.

The job name can be up to 15 characters long and should not contain unusual characters. Set it, along with your project name and the walltime limit on the qsub command line:

qsub -N foo [-l walltime=hhh:mm:ss]  g16.job

Operation

Upon job execution, the script reads your Gaussian input file and modifies it use under PBS on Carbon, as follows:

• Link0-commands %NProcShared=ppn and %LindaWorkers=n456,n457,n458,... are inserted automatically.
• Checkpoint-files named in %chk directives are identified and copied into the compute node's job-specifc $TMPDIR. The %chk specification will be changed to include $TMPDIR. Note that this will be echoed in the Gaussian output as follows:

 %chk=/tmp/12345.sched5.carboncluster/test0420.chk

Chk-file processing will be performed only for files within the current directory; to skip processing, specify a relative or absolute path, such as %chk=./name.chk

• At the end of the job, chk-files in $TMPDIR are moved to $PBS_O_WORKDIR.

Parsing script

Much of the output from Gaussian is available in plaintext format in the log file. Many users write grep-like utilities on their own to extract such information. I (stern (talk)) wrote my own, which is available alongside each Gaussian module.

gauss-parse

For information on usage, run:

gauss-parse --help
gauss-parse --man

To give an idea of scope, here is the beginning of the help output:

Usage:
    gauss-parse [options] [gaussian-output-file ...]

Options:
    --energy    Output energies (default).

    --spectrum  Output eigenvalues.

    --diis      Print DIIS errors.

    --geometry  Output geometries (into separate files unless "-join" is
                given).

    --charge    Output Mulliken charges. When --Spin is used, output spin
                densities instead.

    --force     Output forces.

    --freq      Output frequencies.

    --archive   Expand archive entry.

    --last      Print output for last frame only.

    For energy, geometry, or force modes only:
    …

The script grew from my needs and is not meant as be-all and end-all. Contact me if you enounter a bug or might need additional functionality, or simply find it useful.

Notes on Linda

Gaussian is parallelized in two complementary ways:

• on a single node across its processors using shared memory, and
• across different nodes via a subset of the Linda parallelization language – http://gaussian.com/link0/?tabid=1#Linda

Incomplete node use

Gaussian/Linda may decide not to use the entirety of the nodes made available to it by the preprocess stage in the PBS job file. This section describes how to determine when this is the case and what to do about it.

Consider the following input:

%mem=20mw
#p MP2/6-311G(2df,p) force symm=loose MaxDisk=250000000 test
…

This being run in the job file through the preprocessor produces the input that Gaussian will see, inserting the node names allocated by PBS:

%NProcShared=32
%LindaWorkers=n456,n457
%mem=20mw
#p MP2/6-311G(2df,p) force symm=loose MaxDisk=250000000 test
…

Gaussian will echo its input line by line in its log file:

Entering Gaussian System, Link 0=g16
Initial command:
/opt/apps/gaussian/16-C.01-1/libexec/avx2/g16/l1.exe "/tmp/123456.sched5.carboncluster/Gau-41457.inp" -scrdir="/tmp/123456.sched1.carboncluster/"
Entering Link 1 = /opt/apps/gaussian/16-C.01-1/libexec/avx2/g16/l1.exe PID=     19164.
 
Copyright (c) 1988-2019, Gaussian, Inc.  All Rights Reserved.

This is part of the Gaussian(R) 16 program.  It is based on
the Gaussian(R) 09 system (copyright 2009, Gaussian, Inc.),
the Gaussian(R) 03 system (copyright 2003, Gaussian, Inc.),
…
Cite this work as:
Gaussian 16, Revision C.01,

…
******************************************
Gaussian 16:  ES64L-G16RevC.01  3-Jul-2019
               22-Jan-2025 
******************************************
%NProcShared=32
Will use up to   16 processors via shared memory.
%LindaWorkers=n456,n457
%mem=20mw
SetLPE:  input flags="-v -opt "Tsnet.Node.lindarsharg: ssh" "
SetLPE:    new flags="-v -opt "Tsnet.Node.lindarsharg: ssh"  -nodelist 'n456.carboncluster n457.carboncluster'"
Will use up to    2 processors via Linda.
--------------------------------------------------------------------
#p MP2/6-311G force …
--------------------------------------------------------------------
…

The last line will be specific to your Gaussian route section.

• I highly recommend to use the #p ("profuse") flag to better track progress through Gaussian's link stages.
• The term processors is used inconsistently here. For shared memory, it means cores, but for Linda it means nodes.
• You can track the use of Linda by:

grep exel file.log

• Gaussian may decide to not use Linda after all. A note like this will appear in the log file:

PrsmSu:  requested number of processors reduced to:   6 ShMem   1 Linda.

In this case, decide wether it would be better to resubmit the job than to leave nodes allocated by PBS idle. It is possible that Gaussian's decision is different for different Linda links. Only the longest-running link would be relevant for you to evaluate resubmission.

Error messages

When running jobs on more than one node (i.e., using Linda), you may see messages like these in a job's error stream:

eval server 0 on n456.carboncluster has dropped it's connection.
eval server 0 on n456.carboncluster has dropped it's connection.
subprocess pid = 11505 has exited. status = 0x0000, id = 0, state = 17. command was /opt/..../linda.../opteron …
died after signing in successfully
eval server 0 on n456.carboncluster has dropped it's connection.
…

• These messages can be ignored, according to http://umbc.rnet.missouri.edu/resources/How2RunGAUSSIAN.html :

These messages are not indicative of a problem. They indicate that the Linda work in a Gaussian link is finished, and that Gaussian is continuing with a new link.

• The environment variable GAUSS_FLAGS (as set in the "gaussian" modulefile) is now configured to include the flag -v, to better trace Linda operations on worker nodes. Additional lines like the following will show up in a job's error stream:

ntsnet: using executable file /opt/apps/gaussian/16-C.01-1/libexec/avx2/g16/l302.exel
ntsnet: trying to schedule 1 worker
ntsnet: scheduled a total of 1 worker
ntsnet: starting master process on n456.carboncluster
ntsnet: starting 1 worker on n456.carboncluster

This example shows how one link within a job of #PBS -l nodes=2:ppn=16 gets executed under Linda.