HPC/Carbon Cluster - Development tools

Compilers and development tools

GNU

• GCC
• GDB
• Make
• … other manuals

Intel

We have supported versions of the following commercial tools, currently installed under /opt/intel:

• Fortran
  • Manual for 10.1
  • /opt/intel/{fc,fce}/version
• C++
  • Manual for 10.1
  • /opt/intel/{cc,cce}/version
• Intel Math Kernel Library (MKL), includes Cluster Edition
  • /opt/intel/mkl/version
• VTune analyzer
• Intel Debugger
  • /opt/intel/{idb,idbe}/version

To access the documentation pages, use firefox on a login node:

• /opt/intel/fce/version/doc/Doc_Index.htm
• /opt/intel/cce/version/doc/Doc_Index.htm
• /opt/intel/mkl/version/doc/Doc_index.htm

Note on binary architectures

CPU architecture labels used by Intel and others can be confusing and have changed over time. Here's a summary of the labels and directory names commonly used:

32, ia32, x86

legacy 32-bit x86 generations up to Pentium 4 and Intel Core

em64t, Intel-64, x86_64, AMD-64

64-bit generations of the Xeon branded CPUs, and the Core 2 generation, mostly compatible with AMD's Opteron

64, ia64, Itanium

Itanium – limited ia32 support, but none for x86_64; not used here.

The cce, fce, and idbe directories above provide the 64-bit em64t versions appropriate for the Xeon processors, with ia32 versions being available under cc, fc, idb to compile and run legacy applications. By default, the 64-bit versions are activated in the user's environment ($PATH and others). This is done automatically through /etc/profile.d/{iccvars,ifortvars,mklvarsem64t}.{sh,csh}.

In the future, we will leverage the environment-modules package for version and architecture selection.

Libraries

Intel Math Kernel Library (MKL)

User Notes

• User's Guide
• Reference Manual
• since v10 includes formerly separate Cluster MKL (providing ScaLapack et al.)
• User Guide (pdf) – consult Sec. 5 to resolve linker issues, and 32bit vs. 64bit programming
• provides BLAS, Lapack, ScaLapack, FFTs (including an FFTW interface)
• Detailed linking instructions
• Link example (from /opt/cnm/examples):

 # Sequential:
   -lmkl_intel_lp64 -lmkl_sequential -lmkl_lapack -lmkl_em64t -lmkl_core -lpthread

 # alternative:
   -Wl,--start-group -lmkl_intel_lp64 -lmkl_sequential -lmkl_lapack -lmkl_em64t -lmkl_core -Wl,--end-group \
   -lpthread

 # OpenMP multi-threaded:
   -Wl,--start-group -lmkl_intel_lp64 -lmkl_intel_thread -lmkl_lapack -lmkl_em64t -lmkl_core -Wl,--end-group \
   -lguide -lpthread

• The FFTW interface of the MKL can be linked by prefixing the above options with one of:

-lfftw2xc_intel
-lfftw2xf_intel
-lfftw3xc_intel
-lfftw3xf_intel

Choose the appropriate version between bindings for FFTW v2 vs. v3 and C vs. Fortran.

• Avoid specifying absolute library paths in your Makefile unless you wish to select a specific version. The user environment is normally correctly set up. To force the library version current at the time of writing, add as first flag to the linker options:

-L/opt/intel/mkl/10.0.2.018/lib/em64t (automatically set in $LD_LIBRARY_PATH).

• see HPC/Carbon_Cluster_-_Getting_started#Example_job_file for controlling OpenMP execution.

VTK

• Intro
• VTK install notes

PETSc

• TODO: Portable, Extensible Toolkit for Scientific Computation

MPI

OpenMPI

This is the primary and recommended MPI version. It supports Ethernet and Infiniband interconnect.

• See also: FAQ

MPICH/MPIC2

TBD

Intel MPI

• Documentation

Resource Management

• MOAB cluster suite
  • Documentation overview
  • Torque
  • Gold allocation manager

(description of overlap/differences)