HPC/Applications/impi

Introduction

Intel-MPI is based on MPICH2 and provides an MPICH2 runtime environment for precompiled binaries.

See also

• HPC/Carbon Cluster - Development tools#MPI

Process manager

Use the Hydra process manager (new since impi version 4.x). See sec. 2.4 Scalable Process Management System in the Reference Manual. To do so, simply use mpiexec.hydra instead of the customary mpirun or mpiexec calls:

 mpiexec.hydra \
	-machinefile  $PBS_NODEFILE \
	-np $(wc -l < $PBS_NODEFILE) \
	./a.out

With Hydra, you no longer need to set up and tear down the older and considerably less stable MPD manager.

IMPI vs. OpenMPI

The openmpi and impi modules can coexist at runtime if:

• impi is loaded first and is called using mpiexec.hydra.
• openmpi is loaded second and is called using mpirun.
• binaries using impi are statically linked, i.e.,

ldd binary | grep libmpi.so || echo OK

prints "OK".

• you don't attempt to compile software using the convenience wrappers like mpicc, mpif90, etc. For these wrappers, the last module loaded will be active. It may be possible to use full paths like $IMPI_HOME/bin/mpif90, but this has not been tested fully.

Since openmpi is loaded by default, use the following sequence in ~/.bashrc:

module unload openmpi
module load impi
module load openmpi

Caveat

This replaces possibly specific existing openmpi versions by the default version, which may break packages depending on it.

Work-in-progress: Evaluation if both modules can be loaded by default

• Probably not possible in general, because the following scripts and dynamic libraries overlap:

bin/mpicc
bin/mpicxx
bin/mpiexec
bin/mpif77
bin/mpif90
bin/mpirun
lib/libmpi.so

• Comparison of the names for binaries and scripts:

ls -1 $OMPI_HOME/bin/ > /tmp/ompi
ls -1 $IMPI_HOME/bin/ > /tmp/impi
comm /tmp/?mpi | sort

IMPI only
|       OMPI only
|       |       Both
|       |       |
		mpicc
		mpicxx
		mpiexec
		mpif77
		mpif90
		mpirun
	mpiCC
	mpiCC-vt
	mpic++
	mpic++-vt
	mpicc-vt
	mpicxx-vt
	mpif77-vt
	mpif90-vt
	ompi-clean
	ompi-iof
	ompi-ps
	ompi-server
	ompi_info
	opal_wrapper
	opari
	orte-clean
	orte-iof
	orte-ps
	orted
	orterun
	otfaux
	otfcompress
	otfconfig
	otfdecompress
	otfdump
	otfinfo
	otfmerge
	vtcc
	vtcxx
	vtf77
	vtf90
	vtfilter
	vtunify
IMB-MPI1
compchk.sh
cpuinfo
mpd
mpd.py
mpdallexit
mpdallexit.py
mpdboot
mpdboot.py
mpdcheck
mpdcheck.py
mpdchkpyver.py
mpdcleanup
mpdcleanup.py
mpdexit
mpdexit.py
mpdgdbdrv.py
mpdhelp
mpdhelp.py
mpdkilljob
mpdkilljob.py
mpdlib.py
mpdlistjobs
mpdlistjobs.py
mpdman.py
mpdringtest
mpdringtest.py
mpdroot
mpdrun
mpdrun.py
mpdsigjob
mpdsigjob.py
mpdtrace
mpdtrace.py
mpiexec.hydra
mpiexec.py
mpifc
mpigcc
mpigxx
mpiicc
mpiicpc
mpiifort
mpitune
mpivars.csh
mpivars.sh
mtv.so
pmi_proxy
ptp_impi_proxy.py
tune

• Same for libraries:

ls -1 $OMPI_HOME/lib/ > /tmp/ompi
ls -1 $IMPI_HOME/lib/ > /tmp/impi
comm /tmp/?mpi | sort

		libmpi.so
	libmca_common_sm.la
	libmca_common_sm.so
	libmca_common_sm.so.1
	libmca_common_sm.so.1.0.0
	libmpi.la
	libmpi.so.0
	libmpi.so.0.0.2
	libmpi_cxx.la
	libmpi_cxx.so
	libmpi_cxx.so.0
	libmpi_cxx.so.0.0.1
	libmpi_f77.la
	libmpi_f77.so
	libmpi_f77.so.0
	libmpi_f77.so.0.0.1
	libmpi_f90.la
	libmpi_f90.so
	libmpi_f90.so.0
	libmpi_f90.so.0.0.1
	libopen-pal.la
	libopen-pal.so
	libopen-pal.so.0
	libopen-pal.so.0.0.0
	libopen-rte.la
	libopen-rte.so
	libopen-rte.so.0
	libopen-rte.so.0.0.0
	libopenmpi_malloc.la
	libopenmpi_malloc.so
	libopenmpi_malloc.so.0
	libopenmpi_malloc.so.0.0.0
	libotf.a
	libotf.la
	libvt.a
	libvt.fmpi.a
	libvt.mpi.a
	libvt.omp.a
	libvt.ompi.a
	mpi.mod
	openmpi
libmpi.a
libmpi.so.3.1
libmpi.so.3.2
libmpi.so.4
libmpi.so.4.0
libmpi_dbg.a
libmpi_dbg.so
libmpi_dbg.so.3.1
libmpi_dbg.so.3.2
libmpi_dbg.so.4
libmpi_dbg.so.4.0
libmpi_dbg_mt.a
libmpi_dbg_mt.so
libmpi_dbg_mt.so.3.1
libmpi_dbg_mt.so.3.2
libmpi_dbg_mt.so.4
libmpi_dbg_mt.so.4.0
libmpi_ilp64.a
libmpi_ilp64.so
libmpi_ilp64.so.4
libmpi_ilp64.so.4.0
libmpi_log.a
libmpi_log.so
libmpi_log.so.3.1
libmpi_log.so.3.2
libmpi_log.so.4
libmpi_log.so.4.0
libmpi_log_mt.a
libmpi_log_mt.so
libmpi_log_mt.so.3.1
libmpi_log_mt.so.3.2
libmpi_log_mt.so.4
libmpi_log_mt.so.4.0
libmpi_lustre.so
libmpi_lustre.so.4.0
libmpi_mt.a
libmpi_mt.so
libmpi_mt.so.3.1
libmpi_mt.so.3.2
libmpi_mt.so.4
libmpi_mt.so.4.0
libmpi_panfs.so
libmpi_panfs.so.4.0
libmpi_pvfs2.so
libmpi_pvfs2.so.4.0
libmpich.so
libmpich.so.1
libmpich.so.1.1
libmpichcxx.so
libmpichcxx.so.1
libmpichcxx.so.1.1
libmpigc3.a
libmpigc3.so
libmpigc3.so.3.1
libmpigc3.so.3.2
libmpigc3.so.4
libmpigc3.so.4.0
libmpigc4.a
libmpigc4.so
libmpigc4.so.3.1
libmpigc4.so.3.2
libmpigc4.so.4
libmpigc4.so.4.0
libmpigf.a
libmpigf.so
libmpigf.so.3.1
libmpigf.so.3.2
libmpigf.so.4
libmpigf.so.4.0
libmpigi.a
libmpiic.a
libmpiic.so
libmpiic.so.3.1
libmpiic.so.3.2
libmpiic4.a
libmpiic4.so
libmpiic4.so.3.1
libmpiic4.so.3.2
libmpiif.a
libmpiif.so
libmpiif.so.3.1
libmpiif.so.3.2
libtvmpi.so
libtvmpi.so.3.1
libtvmpi.so.3.2
libtvmpi.so.4
libtvmpi.so.4.0