HPC/Module naming scheme 2016: Difference between revisions
m (migrated conventions to separate page) |
mNo edit summary |
||
| Line 42: | Line 42: | ||
--> | --> | ||
: The modules <code>fftw3</code> and <code>vasp5</code> did ''not'' change name due to more entrenched usage in the package itself, Unix group names, and compilation dependencies. | : The modules <code>fftw3</code> and <code>vasp5</code> did ''not'' change name due to more entrenched usage in the package itself, Unix group names, and compilation dependencies. | ||
== Customizing module selection by OS release == | == Customizing module selection by OS release == | ||
| Line 168: | Line 160: | ||
--> | --> | ||
* Test – same as in [[#Configure in dedicated files|previous section]]. | * Test – same as in [[#Configure in dedicated files|previous section]]. | ||
<!-- | |||
== Dependent modules to be user-loaded == | |||
New-style modules are less implicit (less automatic and less rigid) in loading modules that they depend on. | |||
* Less automatic means that prior to loading a more advanced module you must load all its prerequisites, chosen from the same MPI and (usually) compiler flavor as the advanced module. A missing prerequisite will give errors of the form | |||
… ERROR:151: '''Module''' 'troubled_name' '''depends on one of the module(s)''' 'other_name1 other_name2' … | |||
To resolve this error, edit your <code>~/.bashrc</code> or <code>.modules-1</code> file and add <code>module load …</code> commands for the needed module(s) ''other_names'' before loading "troubled_name". | |||
* Less rigid means that a module does not loads a ''specific'' version of a prerequisite, which gives you, the user, more flexibility in combining modules. | |||
--> | |||
== Minor changes for the ''module'' command == | |||
=== Name completion in command line === | |||
When working interactively in a terminal, you can use a "completion" feature of the Bash shell to complete a partially typed module name and show all names available for the name typed so far. For example: | |||
At a shell prompt (shown as "$"), type: | |||
$ '''module load fft''' | |||
Press the <code><TAB></code> key and the name will be expanded to <code>fftw3/</code> and you'll see two possible completing names, with the cursor waiting at the end of the longest common substring: | |||
$ '''module load fftw3/'''_ | |||
fftw3/3.3/impi-5/intel-16/3.3.4-10 fftw3/3.3/openmpi-1.10/intel-16/3.3.4-11 | |||
fftw3/3.3/intel/3.3.2-1 fftw3/3.3/openmpi-1.4/intel/3.3.2-4 | |||
Type the letter <code>o</code>, hit the <code><TAB></code> key again. The choices will be narrowed down to OpenMPI. | |||
$ '''module load fftw3/3.3/openmpi-1.'''<TAB> | |||
fftw3/3.3/openmpi-1.10/intel-16/3.3.4-11 fftw3/3.3/openmpi-1.4/intel/3.3.2-4 | |||
Typing the digit <code>1</code> will pick the <code>1.'''1'''0</code> MPI version, at which the then remaining single module name choice will be completed, with the cursor waiting after an additional space character: | |||
$ '''module load fftw3/3.3/openmpi-1.10/intel-16/3.3.4-11''' _ | |||
=== "module purge" command === | |||
Previously, the Carbon queueing system was made available to users by modules, | |||
which meant that it was difficult for a user to start with a clean slate, or to make and adjust a custom set of module choices. | |||
You may now safely use the module "purge" command for its intended purpose: | |||
<source lang="bash"> | |||
module purge | |||
</source> | |||
followed by <code>module load …</code> to choose compilers, MPI flavors, and applications. | |||
<!-- | |||
Load the default module set (which varies based on OS release and user choice) as follows: | |||
<source lang="bash"> | |||
module purge | |||
module load defaults/user | |||
</source> | |||
The Intel compilers and OpenMPI continue to be pre-loaded for you. | |||
In other words, without any module customization, you'd see from the <code>module list</code> command: | |||
<source lang="bash"> | |||
module list | |||
</source> | |||
Currently Loaded Modulefiles: | |||
1) intel/16/16.0.1-2 2) openmpi/1.10/intel-16/1.10.2-1 3) defaults/user/2/2.0 | |||
--> | |||
=== Determining default module versions === | |||
The <code>module avail</code> command under CentOS-6 no longer includes the marker <code>"(default)"</code> when one has been set in a <code>.version</code> file. | |||
I am not sure if this is a bug or by design, but the change certainly makes the output more consistent. | |||
To determine which module will be loaded when an abbreviated name is used, I recommend to inspect the first relevant line in the output of one of these commands: | |||
module show ''name'' | |||
module help ''name'' | |||
Revision as of 22:43, February 26, 2016
Introduction
| (*) Environment modules are the means by which software applications are made available to users. Using modules allows users and administrators to pick, by user or even by compute job, a desired or default version of an application from typically several current and historic versions persisting on the system. Older versions are kept to improve reproducibility of results, an important characteristic of scientific computing. |
On Carbon, the naming scheme for environment modules* and the set of default modules being loaded for users has changed. This step was necessary because of increasing diversity and interdependence of applications, their modules, and the underlying operating system.
New-style module names
You may need to adapt the module names that you placed in your shell startup and job files to the new more hierarchical scheme. For most modules, with exceptions show below, the leading name component (the part before any "/") is the same in the old and new naming schemes. What always differs are the name parts after the first slash.
Name change rules
- To use the latest or automatically selected version of a package, remove version numbers from old-style module names of the form
packagename/version, leaving onlypackagename. This is the recommended approach, as you will automatically benefit from future updates and maintenance builds. - To insist on a specific version for a package in new style names:
- Inspect the available flavors and versions (some older modules were not migrated):
module avail packagename.
- Choose the new-style name up to the desired specificity. You may leave out trailing name or directory parts.
- For instance, instead of
vasp5/5.3/openmpi-1.4/intel/5.3.3p3-mkl-3you may writevasp5/5.3/openmpi-1.4orvasp5/5.3, letting the system choose the versions for MPI and compiler that are chosen as defaults at a given time.
- For instance, instead of
- Inspect the available flavors and versions (some older modules were not migrated):
Renamed modules
For the following modules the newer naming convention allowed for and thus uses more consistent names:
OLD NEW
-------------------------------------
asap3 asap/3
ase2 ase/2
ase3 ase/3
g09 gaussian/09
GaussView gaussview (lowercase)
- The modules
fftw3andvasp5did not change name due to more entrenched usage in the package itself, Unix group names, and compilation dependencies.
Customizing module selection by OS release
Configure in .bashrc
To switch over to new-style module names entirely, on both CentOS releases, you could continue using only ~/.bashrc.
To do this:
- Tell CentOS-5 nodes to offer you the new-style module catalog instead of the old one. To do so, simply create an empty customization file:
touch ~/.modules-2
- Then, apply the changes shown in the #Name change rules section above:
vi .bashrc
# or:
nano .bashrc
- Use a text editor of your choice, such as
nanoorvi.
Configure in dedicated files
It is perhaps cleaner to perform the module selection in files dedicated for each CentOS release, so that you can make adjustments independently.
To migrate your existing configuration, manage the following files:
.bashrc .modules-1 .modules-2
Use a helper application to get you started:
modules-migrate
Here is an example of what you might see:
$ modules-migrate This application will help you adapt your environment-modules customizations to Carbon's naming scheme updated for 2016. Your module commands will be moved from .bashrc into files .modules-1 and .modules-2, so that a node's operating system sees only module names in the appropriate scheme. The file text manipulation done here is fairly basic. You will have opportunities to review and edit your changes. Continue? [Y/n] y ====================================================================== diff of changes from .bashrc: ====================================================================== 19,21c19,21 < module load vasp5 < module load g09/D.01.x86_64-2 < module load GaussView --- > # module load vasp5 > # module load g09/D.01.x86_64-2 > # module load GaussView Accept changes in .bashrc? [Y/n] y Putting in place ... `.bashrc.tentative' -> `.bashrc' Opening text editor for .modules-1.tentative .modules-2.tentative ... Continue? [Y/n] y
- (editor session omitted.)
~ ~ ~ ====================================================================== Review: "module" lines in new files: ====================================================================== ==> .modules-1.tentative <== # Carbon modules initialization for v1-style module names, used on CentOS-5. # To stop using v1-style modules, remove this file. # # Initially extracted from .bashrc by modules-migrate . # # This file is in TCL syntax. module load vasp5 module load g09/D.01.x86_64-2 module load GaussView # vim:syntax=tcl: ==> .modules-2.tentative <== # Carbon modules initialization for v2-style module names, used on CentOS-6. # Also used on CentOS-5 if .modules-1 does not exist. # # Initially extracted from .bashrc by modules-migrate . # # This file is in TCL syntax. module load vasp5 module load gaussian/09/D.01.x86_64-2 module load gaussview # vim:syntax=tcl: ====================================================================== Accept changes (a), Edit (e), or Quit (q)? a Renaming files ... `.modules-1.tentative' -> `.modules-1' `.modules-2.tentative' -> `.modules-2' Done.
- Test – same as in previous section.
Minor changes for the module command
Name completion in command line
When working interactively in a terminal, you can use a "completion" feature of the Bash shell to complete a partially typed module name and show all names available for the name typed so far. For example:
At a shell prompt (shown as "$"), type:
$ module load fft
Press the <TAB> key and the name will be expanded to fftw3/ and you'll see two possible completing names, with the cursor waiting at the end of the longest common substring:
$ module load fftw3/_ fftw3/3.3/impi-5/intel-16/3.3.4-10 fftw3/3.3/openmpi-1.10/intel-16/3.3.4-11 fftw3/3.3/intel/3.3.2-1 fftw3/3.3/openmpi-1.4/intel/3.3.2-4
Type the letter o, hit the <TAB> key again. The choices will be narrowed down to OpenMPI.
$ module load fftw3/3.3/openmpi-1.<TAB> fftw3/3.3/openmpi-1.10/intel-16/3.3.4-11 fftw3/3.3/openmpi-1.4/intel/3.3.2-4
Typing the digit 1 will pick the 1.10 MPI version, at which the then remaining single module name choice will be completed, with the cursor waiting after an additional space character:
$ module load fftw3/3.3/openmpi-1.10/intel-16/3.3.4-11 _
"module purge" command
Previously, the Carbon queueing system was made available to users by modules, which meant that it was difficult for a user to start with a clean slate, or to make and adjust a custom set of module choices. You may now safely use the module "purge" command for its intended purpose:
module purge
followed by module load … to choose compilers, MPI flavors, and applications.
Determining default module versions
The module avail command under CentOS-6 no longer includes the marker "(default)" when one has been set in a .version file.
I am not sure if this is a bug or by design, but the change certainly makes the output more consistent.
To determine which module will be loaded when an abbreviated name is used, I recommend to inspect the first relevant line in the output of one of these commands:
module show name module help name