HPC/Applications/python

In recent years, Python-based software distributions with a broad sope became popular. The suites typically contain:

• A Python interpreter.
• The Conda/pip package management systems.
• A wide-ranging set of Python-native packages, extensions, and add-on executables, often including:
  • Idle – Python Integrated DeveLopment Environment
  • iPython and juPyter – Interactive computation system
  • Cython – Compiler for a superset of Python and C/C++
• All libraries and supporting binaries required by the above, e.g.:
  • BLAS/LAPACK – Linear algebra libraries
  • Tcl – Interpreter for the Tool Command Language
  • MPI – Runtime environment for parallel applications using Message Passing Interface, including the mpirun or mpiexec launch commands.

The last item can be problematic because libraries and supporting binaries included in a Python suite can interfere with other software modules on Carbon. For access by users, HPC software typically leverages conventional Unix environment variables like PATH, LD_LIBRARY_PATH, and PYTHONPATH. These environment variables are interpreted front-to-back, which implies priorities and can easily lead to resources from one module overshadowing those from another when they use the same name.

Caveat: Avoid loading modules at the same time for both a Python suite with an included MPI and a separate MPI implementation, especially in your dot-files (.bashrc, .modules-el*).

A Python suite with an included MPI runtime makes parallel computing easily accessible under Python, but this means a "mere python module" can clash with other MPI implementations used on Carbon. Loading a module for a Python suite that contains MPI and a separate MPI module at the same time can be done but then requires more detail for the MPI launcher in job scripts. Without that, non-obvious failures will result for either bundled parallel applications within the Python suite or (worse) unrelated compiled applications on Carbon.

Interpreter-only releases from the Python Software Foundation

Module nomenclature

python/pyMajor.pyMinor/compilerName-compMajor.compMinor/pyMajor.pyMinor.pyPatch-moduleBuild

Contains

• The Python interpreter only, from the python.org sources.

Python interpreters and packages as bundled with the OS

Module nomenclature

python-osname/pyMajor.pyMinor/compilerName-compMajor.compMinor/pyMajor.pyMinor.pyPatch

Contains

• The Python interpreter that comes with the operating system, which is required for many system internals and for system-provided add-on packages (installed via rpm or yum).

These module normally do not need to be loaded. They are included here for visibility in the module system, and as a prerequisite for (usually older) modules which provide Python packages outside of Conda or Pip. Such packages are usually tied to the version of the Python interpreter under which they were installed.

Suites with vendor-distributed package set

Module nomenclature

python-distributor/pyMajor.pyMinor/distMajor/distributor_defined_version[…]-moduleBuild

Contains

• Python software suite with the package selection as distributed by the vendor.

Suites with customized package set

Module nomenclature

python-env-distributor/pyMajor.pyMinor/distMajor/pyVersion[…]-moduleBuild

A default selection of packages deemed useful for jobs on Carbon.

python-env-distributor-purpose/pyMajor.pyMinor/distMajor/pyVersion[…]-moduleBuild

Same, with alternative package selections. (TBD)

Contains

• Python software suite with the package selection customized for Carbon.

This is usually implemented as a Conda "environment" derived from a vendor-distributed Conda suite.