HPC/Applications/python: Difference between revisions

From CNM Wiki
Jump to navigation Jump to search
mNo edit summary
mNo edit summary
Line 1: Line 1:
= Python variants =
{{TOC_right}}
 
== Interpreter-only releases from the Python Software Foundation ==
; Module nomenclature:
: <code>'''python'''/''pyMajor.pyMinor''/''compilerName-compMajor.compMinor''/''pyMajor.pyMinor.pyPatch-moduleBuild''</code>
; Contains:
:* The Python interpreter only, from the [http://www.python.org python.org] sources.
<!-- '''<font color="green">python</font>/<font color="green">''pyMajor.pyMinor''</font>'''/<font color="#c80">''compilerName-compMajor.compMinor''</font>/<font color="#888">''pyMajor.pyMinor.pyPatch''-''carbonBuild''</font> -->
 
== Python interpreters and packages as bundled with the OS ==
; Module nomenclature:
: <code>'''python-''osname'''''/''pyMajor.pyMinor''/''compilerName-compMajor.compMinor''/''pyMajor.pyMinor.pyPatch''</code>
; Contains:
:* The Python interpreter that comes with the operating system, which is required for many system internals and for system-provided add-on packages (installed via rpm or yum).
; Notes:
: Such a module normally does not need to be loaded. It is included here for visibility in the module system, and as a prerequisite for (usually older) modules which provide Python packages outside of Conda or Pip. Such packages are usually tied to the version of the Python interpreter under which they were installed.
 
== Python software suites ==
Within recent years, entire Python software suites became popular,
Within recent years, entire Python software suites became popular,
usually based on the [https://en.wikipedia.org/wiki/Conda_(package_manager) Conda package manager].
usually based on the [https://en.wikipedia.org/wiki/Conda_(package_manager) Conda package manager].
Line 30: Line 13:
** BLAS/LAPACK – Linear algebra libraries
** BLAS/LAPACK – Linear algebra libraries
** MPI – Runtime environment for parallel applications using Message Passing Interface, providing the <code>mpirun</code> or <code>mpiexec</code> commands, plus related binaries and libraries.
** MPI – Runtime environment for parallel applications using Message Passing Interface, providing the <code>mpirun</code> or <code>mpiexec</code> commands, plus related binaries and libraries.


<div style="background: #fdd">
<div style="background: #fdd">
Line 40: Line 22:
</div>
</div>


=== Vendor-distributed suites ===
== Interpreter-only releases from the Python Software Foundation ==
; Module nomenclature:
: <code>'''python'''/''pyMajor.pyMinor''/''compilerName-compMajor.compMinor''/''pyMajor.pyMinor.pyPatch-moduleBuild''</code>
; Contains:
:* The Python interpreter only, from the [http://www.python.org python.org] sources.
<!-- '''<font color="green">python</font>/<font color="green">''pyMajor.pyMinor''</font>'''/<font color="#c80">''compilerName-compMajor.compMinor''</font>/<font color="#888">''pyMajor.pyMinor.pyPatch''-''carbonBuild''</font> -->
 
== Python interpreters and packages as bundled with the OS ==
; Module nomenclature:
: <code>'''python-''osname'''''/''pyMajor.pyMinor''/''compilerName-compMajor.compMinor''/''pyMajor.pyMinor.pyPatch''</code>
; Contains:
:* The Python interpreter that comes with the operating system, which is required for many system internals and for system-provided add-on packages (installed via rpm or yum).
These module normally do not need to be loaded.
They are included here for visibility in the module system,
and as a prerequisite for (usually older) modules which provide Python packages outside of Conda or Pip.
Such packages are usually tied to the version of the Python interpreter under which they were installed.
 
== Vendor-distributed suites ==
; Module nomenclature:
; Module nomenclature:
: <code>'''python-''distributor'''''/''pyMajor.pyMinor''/''distMajor''/''distributor_defined_version[…]-moduleBuild''</code>
: <code>'''python-''distributor'''''/''pyMajor.pyMinor''/''distMajor''/''distributor_defined_version[…]-moduleBuild''</code>
Line 46: Line 45:
:* Python software suite with the '''package selection as distributed by the vendor.'''
:* Python software suite with the '''package selection as distributed by the vendor.'''


=== Suites with customized package selection ===
== Suites with customized package selection ==
<!-- == Custom Conda environments == -->
<!-- == Custom Conda environments == -->
; Module nomenclature:
; Module nomenclature:

Revision as of 17:21, January 10, 2018

Within recent years, entire Python software suites became popular, usually based on the Conda package manager. The suites typically contain:

  • A Python interpreter.
  • The conda/pip package management systems.
  • A wide-ranging set of Python-native packages, extensions, and add-on executables, often including:
    • Idle – Python Integrated DeveLopment Environment
    • iPython and juPyter – Interactive computation system
    • Cython – Compiler for a superset of Python and C/C++
  • Underlying libraries and supporting binaries, e.g.:
    • Tcl – Interpreter for the Tool Command Language
    • BLAS/LAPACK – Linear algebra libraries
    • MPI – Runtime environment for parallel applications using Message Passing Interface, providing the mpirun or mpiexec commands, plus related binaries and libraries.

Caveat: An MPI Runtime within a Python suite makes parallel computing easily accessible under Python, but this means a "mere python module" can clash with other MPI implementations used on Carbon.

Avoid loading modules at the same time for both an MPI implementation and a Python suite, especially in your dot-files. Doing so may lead to non-obvious failures with either the bundled parallel Python applications or (worse) unrelated compiled applications on Carbon.

Interpreter-only releases from the Python Software Foundation

Module nomenclature
python/pyMajor.pyMinor/compilerName-compMajor.compMinor/pyMajor.pyMinor.pyPatch-moduleBuild
Contains
  • The Python interpreter only, from the python.org sources.

Python interpreters and packages as bundled with the OS

Module nomenclature
python-osname/pyMajor.pyMinor/compilerName-compMajor.compMinor/pyMajor.pyMinor.pyPatch
Contains
  • The Python interpreter that comes with the operating system, which is required for many system internals and for system-provided add-on packages (installed via rpm or yum).

These module normally do not need to be loaded. They are included here for visibility in the module system, and as a prerequisite for (usually older) modules which provide Python packages outside of Conda or Pip. Such packages are usually tied to the version of the Python interpreter under which they were installed.

Vendor-distributed suites

Module nomenclature
python-distributor/pyMajor.pyMinor/distMajor/distributor_defined_version[…]-moduleBuild
Contains
  • Python software suite with the package selection as distributed by the vendor.

Suites with customized package selection

Module nomenclature
python-env-distributor/pyMajor.pyMinor/distMajor/pyVersion[…]-moduleBuild
A default selection of packages deemed useful for jobs on Carbon.
python-env-distributor-purpose/pyMajor.pyMinor/distMajor/pyVersion[…]-moduleBuild
Same, with alternative package selections. (TBD)
Contains
  • Python software suite with the package selection customized for Carbon.
This is usually implemented as a Conda "environment" derived from a vendor-distributed Conda suite.
Retrieved from "https://wiki.anl.gov/wiki_cnm/index.php?title=HPC/Applications/python&oldid=6201"