HPC/Applications/impi: Difference between revisions

From CNM Wiki
Jump to navigation Jump to search
m (Stern moved page HPC/Modules/impi to HPC/Applications/impi)
 
(10 intermediate revisions by the same user not shown)
Line 1: Line 1:
__NOTOC__
== Introduction ==
== Introduction ==
Intel-MPI is based on MPICH2 and provides an MPICH2 runtime environment for precompiled binaries.
Intel-MPI is based on MPICH2 and provides an MPICH2 runtime environment for precompiled binaries.
; See also:
* [http://software.intel.com/en-us/articles/intel-mpi-library/ Vendor product overview]
* [[HPC/Carbon Cluster - Development tools#MPI]]
* [http://software.intel.com/en-us/articles/intel-mpi-library-documentation/ Full documentation]


== Process manager ==
== Process manager ==
Line 16: Line 17:


== IMPI vs. OpenMPI ==
== IMPI vs. OpenMPI ==
The [[../#openmpi|'''openmpi''']] and [[../#impi|'''impi''']] modules can '''coexist at runtime''' if:
Since the openmpi module is loaded by default, use the following instead of <code>module load impi</code>:
* impi is loaded first and is called using <code>mpiexec.hydra</code>.
module '''switch''' openmpi impi
* openmpi is loaded second and is called using <code>mpirun</code>.
However, the modules can '''coexist at runtime''' if:
* [[../#impi|'''impi''']] is loaded first and is called using <code>mpiexec.hydra</code>.
* [[../#openmpi|'''openmpi''']] is loaded second and is called using <code>mpirun</code>.
* binaries using impi are ''statically linked'', i.e.,
ldd ''binary-using-impi'' | grep libmpi.so || echo OK
: prints "OK".
* you don't attempt to compile software using the convenience wrappers like <code>mpicc, mpif90,</code> etc. For these wrappers, the last module loaded will be active. It may be possible to use full paths like <code>$IMPI_HOME/bin/mpif90</code>, but this has not been tested fully.
* you don't attempt to compile software using the convenience wrappers like <code>mpicc, mpif90,</code> etc. For these wrappers, the last module loaded will be active. It may be possible to use full paths like <code>$IMPI_HOME/bin/mpif90</code>, but this has not been tested fully.
: Since openmpi is loaded by default, use the following sequence in ~/.bashrc:
: To load impi in addition to keeping openmpi, use the following sequence in ~/.bashrc:
  module unload openmpi
  module unload openmpi
  module load impi
  module load impi
  module load openmpi
  module load openmpi
; Caveat: This replaces possibly specific existing openmpi versions by the default version, which may break packages depending on it.
; Caveat: This replaces possibly specific existing openmpi versions by the default version, which may break packages depending on it.
; Work-in-progress: Evaluation if both modules can be loaded by default.
; Why not load both modules by default?:
: Comparison of the names for binaries and scripts:
: While many files in those distributions differ and thus could coexist, key scripts and libraries use the same name, in particular the following:
  ls -1 $OMPI_HOME/bin/ > /tmp/ompi
  (cd $OMPI_HOME; ls -1 {bin,lib}/* ) > /tmp/ompi
  ls -1 $IMPI_HOME/bin/ > /tmp/impi
  (cd $IMPI_HOME; ls -1 {bin,lib}/* ) > /tmp/impi
  comm /tmp/?mpi | sort
  comm -1 -2 /tmp/?mpi
<pre>
  rm /tmp/?mpi
IMPI only
|      OMPI only
|      |      Both
|      |      |
mpicc
mpicxx
mpiexec
mpif77
mpif90
mpirun
mpiCC
mpiCC-vt
mpic++
mpic++-vt
mpicc-vt
mpicxx-vt
mpif77-vt
mpif90-vt
ompi-clean
ompi-iof
ompi-ps
ompi-server
ompi_info
opal_wrapper
opari
orte-clean
orte-iof
orte-ps
orted
orterun
otfaux
otfcompress
otfconfig
otfdecompress
otfdump
otfinfo
otfmerge
vtcc
vtcxx
vtf77
vtf90
vtfilter
vtunify
IMB-MPI1
compchk.sh
cpuinfo
mpd
mpd.py
mpdallexit
mpdallexit.py
mpdboot
mpdboot.py
mpdcheck
mpdcheck.py
mpdchkpyver.py
mpdcleanup
mpdcleanup.py
mpdexit
mpdexit.py
mpdgdbdrv.py
mpdhelp
mpdhelp.py
mpdkilljob
mpdkilljob.py
mpdlib.py
mpdlistjobs
mpdlistjobs.py
mpdman.py
mpdringtest
mpdringtest.py
mpdroot
mpdrun
mpdrun.py
mpdsigjob
mpdsigjob.py
mpdtrace
mpdtrace.py
mpiexec.hydra
mpiexec.py
mpifc
mpigcc
mpigxx
mpiicc
mpiicpc
mpiifort
mpitune
mpivars.csh
mpivars.sh
mtv.so
pmi_proxy
ptp_impi_proxy.py
tune
</pre>
* Same for libraries:
ls -1 $OMPI_HOME/lib/ > /tmp/ompi
ls -1 $IMPI_HOME/lib/ > /tmp/impi
  comm /tmp/?mpi | sort
<pre>


libmpi.so
bin/mpicc
libmca_common_sm.la
bin/mpicxx
libmca_common_sm.so
bin/mpiexec
libmca_common_sm.so.1
bin/mpif77
libmca_common_sm.so.1.0.0
bin/mpif90
libmpi.la
bin/mpirun
libmpi.so.0
'''lib/libmpi.so'''
libmpi.so.0.0.2
libmpi_cxx.la
libmpi_cxx.so
libmpi_cxx.so.0
libmpi_cxx.so.0.0.1
libmpi_f77.la
libmpi_f77.so
libmpi_f77.so.0
libmpi_f77.so.0.0.1
libmpi_f90.la
libmpi_f90.so
libmpi_f90.so.0
libmpi_f90.so.0.0.1
libopen-pal.la
libopen-pal.so
libopen-pal.so.0
libopen-pal.so.0.0.0
libopen-rte.la
libopen-rte.so
libopen-rte.so.0
libopen-rte.so.0.0.0
libopenmpi_malloc.la
libopenmpi_malloc.so
libopenmpi_malloc.so.0
libopenmpi_malloc.so.0.0.0
libotf.a
libotf.la
libvt.a
libvt.fmpi.a
libvt.mpi.a
libvt.omp.a
libvt.ompi.a
mpi.mod
openmpi
libmpi.a
libmpi.so.3.1
libmpi.so.3.2
libmpi.so.4
libmpi.so.4.0
libmpi_dbg.a
libmpi_dbg.so
libmpi_dbg.so.3.1
libmpi_dbg.so.3.2
libmpi_dbg.so.4
libmpi_dbg.so.4.0
libmpi_dbg_mt.a
libmpi_dbg_mt.so
libmpi_dbg_mt.so.3.1
libmpi_dbg_mt.so.3.2
libmpi_dbg_mt.so.4
libmpi_dbg_mt.so.4.0
libmpi_ilp64.a
libmpi_ilp64.so
libmpi_ilp64.so.4
libmpi_ilp64.so.4.0
libmpi_log.a
libmpi_log.so
libmpi_log.so.3.1
libmpi_log.so.3.2
libmpi_log.so.4
libmpi_log.so.4.0
libmpi_log_mt.a
libmpi_log_mt.so
libmpi_log_mt.so.3.1
libmpi_log_mt.so.3.2
libmpi_log_mt.so.4
libmpi_log_mt.so.4.0
libmpi_lustre.so
libmpi_lustre.so.4.0
libmpi_mt.a
libmpi_mt.so
libmpi_mt.so.3.1
libmpi_mt.so.3.2
libmpi_mt.so.4
libmpi_mt.so.4.0
libmpi_panfs.so
libmpi_panfs.so.4.0
libmpi_pvfs2.so
libmpi_pvfs2.so.4.0
libmpich.so
libmpich.so.1
libmpich.so.1.1
libmpichcxx.so
libmpichcxx.so.1
libmpichcxx.so.1.1
libmpigc3.a
libmpigc3.so
libmpigc3.so.3.1
libmpigc3.so.3.2
libmpigc3.so.4
libmpigc3.so.4.0
libmpigc4.a
libmpigc4.so
libmpigc4.so.3.1
libmpigc4.so.3.2
libmpigc4.so.4
libmpigc4.so.4.0
libmpigf.a
libmpigf.so
libmpigf.so.3.1
libmpigf.so.3.2
libmpigf.so.4
libmpigf.so.4.0
libmpigi.a
libmpiic.a
libmpiic.so
libmpiic.so.3.1
libmpiic.so.3.2
libmpiic4.a
libmpiic4.so
libmpiic4.so.3.1
libmpiic4.so.3.2
libmpiif.a
libmpiif.so
libmpiif.so.3.1
libmpiif.so.3.2
libtvmpi.so
libtvmpi.so.3.1
libtvmpi.so.3.2
libtvmpi.so.4
libtvmpi.so.4.0
</pre>

Latest revision as of 15:21, June 25, 2013

Introduction

Intel-MPI is based on MPICH2 and provides an MPICH2 runtime environment for precompiled binaries.

Process manager

Use the Hydra process manager (new since impi version 4.x). See sec. 2.4 Scalable Process Management System in the Reference Manual. To do so, simply use mpiexec.hydra instead of the customary mpirun or mpiexec calls:

 mpiexec.hydra \
	-machinefile  $PBS_NODEFILE \
	-np $(wc -l < $PBS_NODEFILE) \
	./a.out

With Hydra, you no longer need to set up and tear down the older and considerably less stable MPD manager.

IMPI vs. OpenMPI

Since the openmpi module is loaded by default, use the following instead of module load impi:

module switch openmpi impi

However, the modules can coexist at runtime if:

  • impi is loaded first and is called using mpiexec.hydra.
  • openmpi is loaded second and is called using mpirun.
  • binaries using impi are statically linked, i.e.,
ldd binary-using-impi | grep libmpi.so || echo OK
prints "OK".
  • you don't attempt to compile software using the convenience wrappers like mpicc, mpif90, etc. For these wrappers, the last module loaded will be active. It may be possible to use full paths like $IMPI_HOME/bin/mpif90, but this has not been tested fully.
To load impi in addition to keeping openmpi, use the following sequence in ~/.bashrc:
module unload openmpi
module load impi
module load openmpi
Caveat
This replaces possibly specific existing openmpi versions by the default version, which may break packages depending on it.
Why not load both modules by default?
While many files in those distributions differ and thus could coexist, key scripts and libraries use the same name, in particular the following:
(cd $OMPI_HOME; ls -1 {bin,lib}/* ) > /tmp/ompi
(cd $IMPI_HOME; ls -1 {bin,lib}/* ) > /tmp/impi
comm -1 -2 /tmp/?mpi
rm /tmp/?mpi
bin/mpicc
bin/mpicxx
bin/mpiexec
bin/mpif77
bin/mpif90
bin/mpirun
lib/libmpi.so