HPC/Applications/nwchem

Introduction

• NWChem 6.1 User Documentation

Considerations on Carbon

Memory

• MEMORY

The default memory allocation is far smaller than useful on Carbon. Raise it using, e.g.:

memory total 3200 mb

Disk space

• SCRATCH_DIR

NWChem might use a camparably large amount of disk space in volumes and bandwidth not suitable for your $HOME directory. To avoid quota overruns, use either $SANDBOX or $TMPDIR for NWChem scratch. The *.nw input file is static, which makes it difficult to point to temporary directories via their environment vairables. As a solution, I suggest to create a symbolic link in the job script prior to running NWChem, and use the name of the job script in your *.nw file. See $NWCHEM_HOME/sample.job for the full file. To the following steps:

(1) In the job script, set up a job-specific scratch directory as symlink named "scratch" in the job's working directory. Choose one of:

• $SANDBOX: distributed scratch (not auto-cleaned, high bandwidth but shared file system)
• $TMPDIR: node-local (auto-cleaned, slower)

cat $NWCHEM_HOME/sample.job

#!/bin/bash
#PBS ...
...

dir=$SANDBOX/$PBS_JOBID
#dir=$TMPDIR

link=scratch
mkdir -p $dir
ln -snf $dir $link

mpirun -machinefile $PBS_NODEFILE -np $PBS_NP \
         nwchem foo.nw

# clean sandbox (won't be reached if job terminates for time)
[[ $dir =~ $SANDBOX/* ]] && rm -r $dir

(2) In the NWChem input file, add the line "scratch_dir ./scratch" near the beginning.

cat job.nw

title "foo structure"
echo
scratch_dir ./scratch
geometry units angstroms
...

Trading disk space for CPU

• SEMIDIRECT – Hardware Resource Control

By default, AO integrals are cached on disk. The delays incurred may well be too much for system sizes above several hundred basis functions. It may prove more efficient to recalculate the orbitals as needed, trading disk for CPU:

scf 
 semidirect memsize 800 filesize 0 mb
 ...
end

Thanks to J. Foley for pointing this out.