HPC/Applications/python

There are many Python installations on Carbon, each normally accessed by loading a module as shown below. Each module gives you access to a Python interpreter, one or more Python-native packages for import, and to supporting software. The modules are grouped and named according to the type and scope of the Python installation provided.

To select a particular Python installation, use the shell command module load modulename. As with other modules, each modulename will look like a Unix file name that has one or more directory components. You can (and normally should) use abbreviated module names, with recommended name forms shown below. For abbreviated names, the module shell command will select the most suitable module from the modules that would complete your abbreviated name. This will usually be the module with the highest version number or an administrator-designated default.

What follows are the types of Python installations and their module naming convention. For the current full list of these modules, run the shell command module avail python.

Python standard distributions

Module nomenclature

python/pyMajor.pyMinor/compilerName-compMajor.compMinor/pyMajor.pyMinor.pyPatch-moduleBuild

Contains

A Python interpreter and solely its standard packages, as distributed from python.org.

Examples

python/2.7/gcc-4.1/2.7.3-1

python/2.7/gcc-4.4/2.7.11-1

python/3.5/gcc-4.4/3.5.1-1

Recommend module name form

python/2.7

python/3.5

Use one of these modules as base for installing one or a few Python-native packages yourself, as long as the dependencies are not too odious or performance is not critical. For more complex requirements, choose one of the Conda-based suites shown below.

Caution: Packages that were installed under one Python version (or module) usually are not accessible through another version, especially packages that contain binaries like shared libraries and executables.

OS-provided bundles

Module nomenclature

python-osname/pyMajor.pyMinor/compilerName-compMajor.compMinor/pyMajor.pyMinor.pyPatch

Contains

• The Python interpreter that comes with the operating system. It is required for many system internals and is the version expected by system-provided add-on packages (installed via rpm or yum).

Examples

python-centos/2.4/gcc-4.1/2.4.3

python-centos/2.6/gcc-4.4/2.6.6

python-centos/2.7/gcc-4.8/2.7.5

Recommend module name form

(none)

You normally do not need to load these modules. They are included here for visibility in the module system, and as a prerequisite for (usually older) modules which provided Python independently (instead of by package manager), and which are usually tied to the version of the Python interpreter under which they were installed.

Suites with vendor-distributed package set

Module nomenclature

python-distributor/pyMajor.pyMinor/distMajor/distributor_defined_version[…]-moduleBuild

Contains

Python software suite with the package selection as distributed by the vendor.

Examples

python-intel/2.7/2018/2.7.14-2018.1.023-1

python-intel/3.5/2017/3.5.3-2017.3.052-1

python-intel/3.6/2018/3.6.3-2018.1.023-1

python-anaconda/2.7/4/2.7.11-4.0.0-2

Recommend module name form

python-intel/2.7

python-intel/3.6

python-anaconda/2.7

python-anaconda/3.5

Python-based software suites with a broad sope became popular in recent years. They typically contain:

• A Python interpreter.
• The Conda/pip package management systems.
• A wide-ranging set of Python-native packages, extensions, and add-on executables, often including:
  • NumPy/SciPy – Ecosystem for scientific computing in Python
  • Idle – Python Integrated DeveLopment Environment
  • iPython and juPyter – Interactive computation system
  • Cython – Compiler for a superset of Python and C/C++
• All libraries and supporting binaries required by the above, e.g.:
  • BLAS/LAPACK – Linear algebra libraries
  • Tcl – Interpreter for the Tool Command Language
  • MPI – Runtime environment for parallel applications using Message Passing Interface, including the mpirun or mpiexec launch commands.

The last item can be problematic because libraries and supporting binaries included in a Python suite can interfere with other software modules on Carbon. For access by users, HPC software typically leverages conventional Unix environment variables like PATH, LD_LIBRARY_PATH, and PYTHONPATH. These environment variables are interpreted front-to-back, which implies priorities and can easily lead to resources from one module overshadowing those from another when they use the same name.

In particular, a Python suite with an included MPI runtime makes parallel computing easily accessible under Python, but this means a "mere python module" can clash with other MPI implementations used on Carbon. Loading a module for a Python suite that contains MPI and a separate MPI module at the same time can be done but then requires more detail for the MPI launcher in job scripts. Without that, non-obvious failures will result for either bundled parallel applications within the Python suite or (worse) unrelated compiled applications on Carbon.

Caution: Avoid loading modules at the same time for both a Python suite with an included MPI and a separate MPI implementation, especially in your dot-files (.bashrc, .modules-el*).

Various workarounds for MPI clashes exist:

• Altering the module load order.
• Launching the non-python MPI variant with qualified paths like $FOO_HOME/bin/mpirun.
• If MPI is not needed on the Python side, use a custom virtual environment that has MPI excluded (described in the next section).

Please contact [email protected] for help to choose an approach.

Suites with customized package set

Module nomenclature

python-env-distributor/pyMajor.pyMinor/distMajor/pyVersion[…]-moduleBuild

Python with a default selection of packages deemed useful for jobs on Carbon.

python-env-distributor-purpose/pyMajor.pyMinor/distMajor/pyVersion[…]-moduleBuild

Same, with alternative package selections. (TBD)

Contains

Python virtual environment with a package selection customized for Carbon.

The virtual environments are derived from one of the vendor-distributed Conda suites described in the previous section.