HPC/Module naming scheme 2016

Introduction

(*) Environment modules are the means by which software applications are made available to users. Using modules allows users and administrators to pick, by user or even by compute job, a desired or default version of an application from typically several current and historic versions persisting on the system. Older versions are kept to improve reproducibility of results, an important characteristic of scientific computing.

On Carbon, the environment modules* system has changed in the following aspects, all explained further in this document:

• The naming scheme is more developed and more versatile.
• Default and dependent modules are no longer being loaded.
• The module command behaves in slightly different ways.

Motivation

The changes were necessary because of increasing diversity and interdependence of applications, their modules, and the underlying operating system. The goal was to accommodate different compilers, MPI flavors, and (in the future) different aspects of the machine architecture like CPU generation, capabilities, and coprocessor facilities.

Further, for different OS releases the new scheme enables existing application versions to continue being offered where possible, and to make new application versions available where suitable, either on both old and newer OS releases, or only on one.

Naming scheme (Nomenclature)

The general module naming scheme is as follows:

• The full name of a module has several components that are separated by a slash /.
• The first and last component of the module name, respectively, are formed by the applications's typically author-provided main name and version, along with a build number or identifier that is local to Carbon.
• Other name components may be present in-between and make apparent to the user which set of major tools was used to produce the application locally, which usually translates to which modules must be loaded to run the application.

In detail, module names have one of the following forms and typical use cases:

name/api/version-build			# binary packages, compilers
name/api/compiler/version-build		# compiled applications
name/api/mpi/compiler/version-build	# compiled applications that use MPI

• name is the package's name as chosen on Carbon. It is usually the name given by the software's author, but lowercased for consistency, and it may contain a number if conventionally so named by the author, e.g. fftw3.
• api is the leading part or parts of the package's version number which typically signifies to suitable precision the API level across which different package versions are expected to be compatible (interchangable in terms of features). api is typically a sole major version number, or has the form major.minor. You may load a module that has the full name foo/m.n/compiler/version-build by the abbreviated name foo/m.n/compiler, which enables you to select the features and binary compatibility level that you need without having to give a complete name all the way down to a build number.
• compiler is a name component that is present when an application was compiled here and thus usually needs runtime libraries associated with the compiler used. The compiler name component is not strictly needed for applications that are statically linked, but is usually present even then for informative purposes. Conversely, the name component is typically not present for applications installed from binary distribution packages, notably commercial applications and, naturally, compilers themselves.
• mpi, present when neeeded, denotes the MPI flavor in use for parallel computations.

For reference and contrast, the previous scheme has been the simpler but more opaque:

name/version-build

Example: Names for the FFT3 library module in the old and new naming schemes, queried by the module avail shell command:

Scheme 2 (current)	Scheme 1 (being retired)
$ module -t avail fftw3 /opt/apps/M/x86_64/EL6: /opt/apps/M/x86_64/EL: fftw3/3.3/impi-5/intel-16/3.3.4-10 # uses Intel-MPI fftw3/3.3/intel/3.3.2-1 # legacy, serial only fftw3/3.3/openmpi-1.10/intel-16/3.3.4-11 # uses OpenMPI fftw3/3.3/openmpi-1.4/intel/3.3.2-4 # legacy, OpenMPI /usr/share/Modules/modulefiles: /etc/modulefiles:	$ module -t avail fftw3 /opt/soft/modulefiles: fftw3/3.2.1-1 fftw3/3.2.2-1 fftw3/3.3-1 fftw3/3.3.2-1 fftw3/3.3.2-4 /opt/intel/modulefiles: /usr/share/Modules/modulefiles: /etc/modulefiles:

• Note the MPI flavor and the compiler name components compared to the older naming scheme. (Boldface shown here for illustration only, your output will appear all as regular text.)
• The -t option of module avail shows the output in "terse" form, one entry per line.
• Lines ending in : indicate file system directories in which modules are being located on the current node.

Watch out: Changes requiring your attention

You may need to adapt the modules that you load in your shell startup and job files to the new naming scheme.

Follow the Migration guide section below to edit your affected files. Change the affected module names as follows:

Name changes for most modules

For most modules, with a few exceptions, the leading name component (the part before any "/") is the same in the old and new naming schemes. What always differs are the name parts after the first slash, mostly relevant if you previously chose a specific version deliberately and hopefully with good reason.

• To select the latest or administratively designated default version of a package:
  1. On a Carbon command line, list the available flavors and versions, keeping in mind that some older modules were not migrated:

     module avail packagename

  2. In your configuration file, remove version numbers from old-scheme module names of the formpackagename/version, leaving only packagename.
  3. Append API, MPI, and/or compiler specifications as needed.

  This is the recommended approach, as you will automatically benefit from future updates and maintenance builds.

  For instance, instead of vasp5/5.3/openmpi-1.4/intel/5.3.3p3-mkl-3 write vasp5/5.3/openmpi-1.4

• To insist on a specific version and build for a package in new-scheme names, which you should do only if you require a build with a specific feature or behavior:

4. Append version and build specifications, as shown by the module avail packagename command above.

Name change exceptions

For the following modules the newer naming convention allowed for and thus uses more consistent names:

OLD		NEW
-------------------------------------
asap3		asap/3.x		

ase2		ase/2		- deprecated
ase3		ase/3		- not needed as separate module, instead, is installed within each of the new "python-env" modules

g09		gaussian/09
GaussView	gaussview  (lowercase)

python		python-env	- Several suites of Python environments, each with many packages
		python.org	- The interpreter only, from the main Python web site.

Note that the modules fftw3 and vasp5 did not change name due to more entrenched usage of their qualified name in the packages themselves, in Unix group names, and in makefiles.

Scheme customization by operating system

During a transition period, nodes with different operating systems will coexist in the cluster. Each user, nonetheless, will see the same networked home directory on each node regardless of the OS that the node runs. Therefore, the user's shell and module configuration files (like .bashrc and others) will be interpreted by system utilities, end user applications, and in runtime environments that all differ by operating system, to a varying degree.

The module scheme that is active on a node is primarily determined by its operating system, secondarily by the user, according to these rules:

• On nodes running CentOS-6, scheme 2 is used always. Customize in ~/.bashrc or in a separate configuration file ~/.modules-2.
• On CentOS-5 nodes, scheme 1 is normally used. If only the file ~/.modules-2 exists, scheme 2 is used instead. You can go back to scheme 1 on these nodes by creating ~/.modules-1.

In other words, the files will be detected and read as follows:

You have files: .bashrc and …		Remark	CentOS-5 reads:		CentOS-6 reads:		CentOS-7 reads:
			files	module names	files	module names	files	module names
–	–	Starting situation	.bashrc only	scheme 1	.bashrc only	scheme 2	To be determined.
–	.modules-2	Full switch-over	.modules-2 and .bashrc	scheme 2	.modules-2 and .bashrc
.modules-1	.modules-2	Separate configs, recommended	.modules-1 and .bashrc	scheme 1	.modules-2 and .bashrc

Remember that your home directory is the same across nodes, and therefore your configuration scripts are sensitive to differences between OS releases.

Available modules differ between naming schemes

• The previous module naming scheme 1 is being retired, along with some of its attendant conventions.
• Newer applications will primarily be compiled and installed on the newer OS release and in naming scheme 2. Some applications may turn out to be backwards-compatible with a previous OS release, and will be made available there as well, in scheme 2, to appropriately offer applications that run on both or only a specific release of the operating system.
• Only a subset of modules from scheme 1 has been ported to scheme 2, typically the modules representing the most recent version of an application.

New modules do not implicitly load their dependencies

• A module from the new scheme does not implicitly load other modules that it might depend on, such as modules for compilers, an MPI flavor, or specialized libraries. Previously, this was the case for some popular modules but with the system maturing and diversifying, the side effects of such behind-the-scenes actions can lead to trouble.
• Learn to recognize error messages that are issued by the module load command when it notices that a dependent module is not loaded.

Example: A typical error message looks like:

$ module load openmpi/1.10
openmpi/1.10/intel-16/1.10.2-1(27):ERROR:151: Module 'openmpi/1.10/intel-16/1.10.2-1' depends on one of the module(s) 'intel/16/16.0.2 intel/16/16.0.1-2 intel/16/16.0.0-3 intel/16/16.0.0-1 intel/16/16.0.0-0'
openmpi/1.10/intel-16/1.10.2-1(27):ERROR:102: Tcl command execution failed: prereq intel/16

Colors do not appear in the original terminal output but were added here for clarity:

• The missing prerequisite is the red item on the last line.
• The modules that would currently satisfy the requirement are shown on the preceding line, indicated here in blue.
• The full name of the "offending module", deduced from a possibly abbreviated name on the command line, appears in brown.

• Resolve your module dependencies by augmenting the module load statements in your configuration files. While this may by a minor burden for you at first, your selections will become easier to understand now and easier to adapt later. Load prerequisite modules before you load modules that depend on them, by full or (recommended) abbreviated name. The typical order is: compiler, MPI, other (dynamic) libraries, and finally your intended module. For the example shown, suitable commands would be:

module load intel/16
module load openmpi/1.10/intel-16

Note also how in this case it is preferable to explicitly supply the compiler name component …/intel-16 of the openmpi module to ensure it is consonant with the compiler.

You must load all modules yourself

(**) Technically, the system does load the module profile/user for you. This module only contains the instructions to select and read your .modules-1 or .modules-2 file.

• No modules are pre-loaded by the system**. Previously, the Intel compilers, MKL, and OpenMPI were pre-loaded for you.
• Now, you must yourself load all desired modules, particularly compiler and MPI modules, in your shell setup files or in job files (see section below), in suitable order. Modules not depending on others must be loaded first.

This is born of necessity because still useful older applications were compiled with older MPI flavors and versions (typically OpenMPI-1.4) which partially interfere with newer flavors (OpenMPI-1.8, 1.10, or Intel-MPI-5.x). In particular, each MPI flavor provides commands like mpirun and mpifort, and special care is needed to run the correct one if your chosen module set spans different MPI flavors.

• To determine which modules are required, first select a high-level module, inspect the output of module show name, and look for any prereq statements. When editing your .modules-* file, load the prereq modules first, then the high-level module that you want.

Example:

• Let's load a vasp5 module that uses the Intel-MPI flavor, named "impi":

$ module avail vasp5
------------------------------------------------------------ /opt/apps/M/x86_64/EL ------------------------------------------------------------
vasp5/5.3/openmpi-1.4/intel/5.3.2-mkl-beef-1    vasp5/5.4/impi-5/intel-16/5.4.1.3-6
vasp5/5.3/openmpi-1.4/intel/5.3.3p3-mkl-3       vasp5/5.4/openmpi-1.10/intel-16/5.4.1.3-6
vasp5/5.3/openmpi-1.4/intel/5.3.3p3-mkl-cellz-1

• Let's try inspecting what it needs:

$ module show vasp5/5.4
-------------------------------------------------------------------
/opt/apps/M/x86_64/EL/vasp5/5.4/openmpi-1.10/intel-16/5.4.1.3-6: 
…

• Careful now: The first output line shows the full file name of the module that would get loaded by the short name. In this case, the abbreviated module name, having no MPI name component, yields a module that uses a different MPI flavor than you want.
• You will need to be more explicit:

$ module show vasp5/5.4/impi-5
-------------------------------------------------------------------
/opt/apps/M/x86_64/EL/vasp5/5.4/impi-5/intel-16/5.4.1.3-6:

module-whatis	 VASP - Vienna Ab-initio Simulation Package 
conflict	 vasp 
conflict	 vasp-vtst 
prereq	 intel/16
prereq	 impi/5
setenv		 VASP5_HOME /opt/apps/vasp5/5.4.1.3-6-impi-5-intel-16 
prepend-path	 PATH /opt/apps/vasp5/5.4.1.3-6-impi-5-intel-16/bin 
setenv		 VASP_COMMAND vasp-ase 
setenv		 VASP_PP_PATH /opt/soft/vasp-pot/ase 
-------------------------------------------------------------------

• Therefore, you'd need to add the following lines to your .modules-2 file:

module load intel/16
module load impi/5
module load vasp5/5.4/impi-5

Expert Tip

The grep command does not work as usual on module show … because of the way module needs to operate. To make grep work, combine the stdout and stderr streams using, in bash, the |& characters to form the pipe:

$ module show vasp5/5.4/impi-5 |& grep prereq
prereq	 intel/16
prereq	 impi/5

Loading modules in job files

• You may now safely load modules in PBS job files when using recent MPI modules. Previously, this was not recommended.

Recent builds of OpenMPI (1.4 and 1.10) and Intel MPI now have support compiled in to properly start proccesses on remote nodes.

• However, best practice is to load all modules in ~/.bashrc or ~/.modules-2.

This will always give you the same applications on both login and compute nodes. Place module commands in job files only when conflicts arise, such as when two of your regularly-used applications require different MPI flavors.

Multiple MPI Flavors

• Different MPI flavors can, with caution, be loaded at the same time. This may be necessary because the system is less homogeneous than in the past and no longer uses a single "one true" MPI implementation.
• When modules of multiple MPI flavors are loaded, call the appropriate MPI commands by a full path specified via the MODULENAME_HOME environment variables that is set by the modules.

Example: In a job file that is to run 2 applications that were compiled with different MPI flavors, write:

$OPENMPI_HOME/bin/mpirun app1_name
$IMPI_HOME/bin/mpirun app2_name

Migration guide – How to customize your module selection by OS release

In adapting your existing module selection, you have two choices:

Configure in dedicated files

It is cleanest to perform the module selection in separate files, one for each OS release. This allows you to change files later on with minimal interference.

• To migrate your existing configuration, use a helper utility to get you started:

modules-migrate

This utility will manage the following files, which you should afterwards inspect and edit:

.bashrc
.modules-1
.modules-2

See example output of running the helper.

• Afterwards, test your module choices.

Configure in .bashrc

To switch over to new-style module names entirely, on both CentOS releases, you could continue using only ~/.bashrc. To do this:

• Tell CentOS-5 nodes to offer you the new-style module catalog instead of the old one. To do so, simply create the new-scheme customization file but leave it empty:

touch ~/.modules-2

• Then, apply the changes shown in the nomenclature section above:

vi .bashrc
# or:
nano .bashrc

Use a text editor of your choice, such as nano or vi.

• Test – see next section.

Test your module choices

Same node	EL5 node	EL6 node
bash -l	ssh clogin5	ssh clogin7
module list
exit

To test your new module configuration:

1. Open another login shell on the current or another node.
2. Review error messages that might appear before your prompt.
3. Inspect which modules are loaded.
4. Edit your .module-* files to mitigate any errors.
5. Close the testing shell and repeat.

Minor changes for the module command

Determining default module versions

To determine which module will be loaded when an abbreviated name is used, inspect the first relevant line in the output of one of these commands:

module show name
module help name

The reason is twofold:

• The module avail command under CentOS-6 no longer issues the marker "(default)" when set for a particular module (which is done administratively using a .version file). I am not sure if this is a bug or by design, but the change makes the output more consistent.
• On the older CentOS-5 system the module command honors .version files only for the last component of the module. This may lead to different module versions being selected on different systems even when the list of available modules is identical. (Side note: This is a possibly fortuitous bug since openmpi-1.4, used on CentOS-5, sorts after openmpi-1.10.)

Name completion on command line

When working interactively in a terminal, you can use the "Tab completion" feature of the Bash shell to complete a partially typed module name and show all names available for the name typed so far.

The feature works as follows. At a shell prompt (shown as "$"), type:

$ module load fft

Press the <TAB> key and the name will be expanded to fftw3/3.3/, and you'll see all possible completing names, with the cursor waiting at the end of the longest common substring:

$ module load fftw3/3.3/_
fftw3/3.3/impi-5/intel-16/3.3.4-10        fftw3/3.3/openmpi-1.10/intel-16/3.3.4-11
fftw3/3.3/intel/3.3.2-1                   fftw3/3.3/openmpi-1.4/intel/3.3.2-4

Type the letter o, hit the <TAB> key again. The choices will be narrowed down to OpenMPI.

$ module load fftw3/3.3/openmpi-1.<TAB>
fftw3/3.3/openmpi-1.10/intel-16/3.3.4-11  fftw3/3.3/openmpi-1.4/intel/3.3.2-4

Typing the digit 1 will pick the 1.10 version, at which point the then remaining single module name choice will be completed all the way, with the cursor waiting after an additional space character:

$ module load fftw3/3.3/openmpi-1.10/intel-16/3.3.4-11 _

"module purge" command

Previously on Carbon it was difficult to reset the module selection during an interactive terminal session, because the commands for the job queueing system, like qsub, were provided via a module. You may now safely use the module "purge" command for its intended purpose, as

module purge

followed by module load … to choose compilers, MPI flavors, and applications.

Expert Tip: Purge and reload.

You can re-load the customizations from your .modules-1 or .modules-2 files using the module profile:

module purge
module load profile