Job Submission and Monitoring: Difference between revisions
Jump to navigation
Jump to search
No edit summary |
|||
| Line 34: | Line 34: | ||
p011, p012 | p011, p012 | ||
================================================================================ | ================================================================================ | ||
< | </PRE> | ||
==Jobs Submission Basics== | ==Jobs Submission Basics== | ||
Revision as of 04:30, November 20, 2025
Resource Summary View (qsum)
$ qsum
=============== A4000 ==========================================================
A4000 / nodes: 3 / down: 0 / offline: 0 / busy: 0 / available: 3
AVAILABLE (3): g001, g002, g003
=============== A6000 ==========================================================
A6000 / nodes: 1 / down: 0 / offline: 0 / busy: 0 / available: 1
AVAILABLE (1): lambda01
=============== AMD16 ==========================================================
AMD16 / nodes: 33 / down: 2 / offline: 0 / busy: 2 / available: 29
DOWN (2): n017, n030
ley (2): n001, n002
AVAILABLE (29): n003, n004, n005, n006, n007, n008, n009, n010, n011, n012
n013, n014, n015, n016, n018, n019, n020, n021, n022, n023
n024, n025, n026, n027, n028, n029, n031, n032, n039
=============== EPYC1 ==========================================================
EPYC1 / nodes: 1 / down: 0 / offline: 0 / busy: 0 / available: 1
AVAILABLE (1): a027
=============== EPYC2 ==========================================================
EPYC2 / nodes: 20 / down: 0 / offline: 0 / busy: 5 / available: 15
ley (2): a030, a031
msitek (3): a028, a029, a032
AVAILABLE (15): a033, a034, a035, a036, a037, a038, a039, a040, a041, a042
a043, a044, a045, a046, a047
=============== VIRTUAL ========================================================
VIRTUAL / nodes: 6 / down: 0 / offline: 0 / busy: 0 / available: 6
AVAILABLE (6): v001, v002, v003, v004, v005, v006
=============== XEON28 =========================================================
XEON28 / nodes: 12 / down: 0 / offline: 0 / busy: 0 / available: 12
AVAILABLE (12): p001, p002, p003, p004, p005, p006, p007, p008, p009, p010
p011, p012
================================================================================
Jobs Submission Basics
Sample LS-Dyna job submission script looks like this:
#!/bin/bash # #PBS -q extra #PBS -A ls-dyna #PBS -l nodes=2:ppn=64 #PBS -l walltime=100:: #PBS -j oe #PBS -N male_female #PBS -e log.error #PBS -o log.output #PBS -W x=NACCESSPOLICY:SINGLEJOB #PBS -m bae module load ls-dyna/MPP/SP/AVX2/12.2.1 cd $PBS_O_WORKDIR mpirun -machinefile $PBS_NODEFILE ls-dyna i=main.k memory1=300m memory2=100m module load dynamore/evaluation DM.plotcprs.lnx -merge
For an interactive session that makes it easier to troubleshoot your jobs during development, you can allocate machines for your job using the following syntax:
qsub -I -q extra -l nodes=2:ppn=64,walltime=24:00:00
LS-Dyna on the ARROW Cluster
Currently Available LS-Dyna Versions
The following is a list of LS-Dyna versions available on ARROW after the latest reconfiguration of the system. Versions below 12.0.0 are no longer available because they are not compatible with modern operating systems and cannot be made to work correctly.
All versions are loaded using the "module load" command. Versions can be listed with the "module avail ls-dyna" command. To load opne of the modules, use the following syntax:
module load ls-dyna/14.2.0/mpi-d8-ifort190-avx512
The version string is composed of multiple elements to indicate variants in compilers and compiler options. Use the following guideline to choose an appropriate version to load:
- "1" or "mpi" indicates whether this is a single node version of LS-Dyna or whether this is a multi-node MPI version. All MPI versions use the IntelMPI 2022 libraries which have been tested thoroughly on ARROW. MPI versions will use the Infiniband Network of ARROW for high-speed and low-latency inter-process communication using RDMA (remote direct memory access).
- All LS-Dyna versions are available in either floating point or double precision variants. Floating point variants use 4 bytes to represent a value, and double precision variants use 8 bytes. There are pros and cons for choosing one over the other variant. With regards to computational efficiency, both perform nearly the same because all machines are equipped with 64-bit CPUs.
- "f4" floating point versions
- Pros: These require significantly less memory to run. Results occupy less disk space, and can be transferred significantly faster into and out of ARROW.
- Cons: The numerical resolution is limited to 7 significant digits, which is often undesirable when dealing with mathematical operations on small and large numbers at the same time.
- "r8" double precision versions
- Pros: The numerical resolution is about twice the number of significant digits compare to "f4", which helps when when dealing with mathematical operations on small and large numbers at the same time.
- Cons: These require more memory to run. Results occupy more disk space, and it takes longer to transfer data into and out of ARROW.
- "f4" floating point versions
- There are two more identifiers to choose from when it comes to the variants of the executables: the specific compiler used to create the executable and the specific processor instruction set required for running the executable.
- For modern versions of LS-Dyna, two compilers have been used by the developers to create LS-Dyna executables: the Intel Fortran Compiler and the AOCC (AMD Optimizing C/C++ and Fortran) compiler. Both variants of the software are supported on ARROW. This gives users the opportunity to choose an alternate variant of the same LS-Dyna version when running into bugs or crashes.
- The variants based on the various instruction set extensions (SSE2, AVX2, AVX512, and so on) gives users even more options when choosing an alternate LS-Dyna variant of the same version when running into bugs or crashes. These instruction sets are mostly related to performance gains on specific processors. We have not performed thorough performance tests and cannot recommend specific versions right now.
$ module avail ls-dyna ------------------------------------- /shared/apps/modulefiles ------------------------------------- ls-dyna/12.1.0/1-d8-aocc310 ls-dyna/14.1.0/mpi-d8-aocc400-avx2 ls-dyna/12.1.0/1-d8-ifort160 ls-dyna/14.1.0/mpi-d8-ifort190-avx2 ls-dyna/12.1.0/1-f4-aocc310 ls-dyna/14.1.0/mpi-d8-ifort190-avx512 ls-dyna/12.1.0/1-f4-ifort160 ls-dyna/14.1.0/mpi-d8-ifort190-sse2 ls-dyna/12.1.0/mpi-d8-aocc310-avx2 ls-dyna/14.1.0/mpi-f4-aocc400-avx2 ls-dyna/12.1.0/mpi-d8-ifort160-avx2 ls-dyna/14.1.0/mpi-f4-ifort190-avx2 ls-dyna/12.1.0/mpi-d8-ifort160-sse2 ls-dyna/14.1.0/mpi-f4-ifort190-avx512 ls-dyna/12.1.0/mpi-f4-aocc310-avx2 ls-dyna/14.1.0/mpi-f4-ifort190-sse2 ls-dyna/12.1.0/mpi-f4-ifort160-avx2 ls-dyna/14.2.0/1-d8-aocc400-avx2 ls-dyna/12.1.0/mpi-f4-ifort160-sse2 ls-dyna/14.2.0/1-d8-ifort190-sse2 ls-dyna/12.2.0/1-d8-aocc400 ls-dyna/14.2.0/1-f4-aocc400-avx2 ls-dyna/12.2.0/1-d8-ifort160 ls-dyna/14.2.0/1-f4-ifort190-sse2 ls-dyna/12.2.0/1-f4-aocc400 ls-dyna/14.2.0/mpi-d8-aocc400-avx2 ls-dyna/12.2.0/1-f4-ifort160 ls-dyna/14.2.0/mpi-d8-ifort190-avx2 ls-dyna/12.2.0/mpi-d8-aocc400-avx2 ls-dyna/14.2.0/mpi-d8-ifort190-avx512 ls-dyna/12.2.0/mpi-d8-ifort160-avx2 ls-dyna/14.2.0/mpi-d8-ifort190-sse2 ls-dyna/12.2.0/mpi-d8-ifort160-sse2 ls-dyna/14.2.0/mpi-f4-aocc400-avx2 ls-dyna/12.2.0/mpi-f4-aocc400-avx2 ls-dyna/14.2.0/mpi-f4-ifort190-avx2 ls-dyna/12.2.0/mpi-f4-ifort160-avx2 ls-dyna/14.2.0/mpi-f4-ifort190-avx512 ls-dyna/12.2.0/mpi-f4-ifort160-sse2 ls-dyna/14.2.0/mpi-f4-ifort190-sse2 ls-dyna/12.2.1/1-d8-aocc400-avx2 ls-dyna/15.0.2/1-d8-aocc400-avx2 ls-dyna/12.2.1/1-d8-ifort160-sse2 ls-dyna/15.0.2/1-d8-ifort190-sse2 ls-dyna/12.2.1/1-f4-aocc400-avx2 ls-dyna/15.0.2/1-f4-aocc400-avx2 ls-dyna/12.2.1/1-f4-ifort160-sse2 ls-dyna/15.0.2/1-f4-ifort190-sse2 ls-dyna/12.2.1/mpi-d8-aocc400-avx2 ls-dyna/15.0.2/mpi-d8-aocc400-avx2 ls-dyna/12.2.1/mpi-d8-ifort160-avx2 ls-dyna/15.0.2/mpi-d8-ifort190-avx2 ls-dyna/12.2.1/mpi-d8-ifort160-sse2 ls-dyna/15.0.2/mpi-d8-ifort190-avx512 ls-dyna/12.2.1/mpi-f4-aocc400-avx2 ls-dyna/15.0.2/mpi-d8-ifort190-sse2 ls-dyna/12.2.1/mpi-f4-ifort160-avx2 ls-dyna/15.0.2/mpi-f4-aocc400-avx2 ls-dyna/12.2.1/mpi-f4-ifort160-sse2 ls-dyna/15.0.2/mpi-f4-ifort190-avx2 ls-dyna/12.2.2/1-d8-aocc400-avx2 ls-dyna/15.0.2/mpi-f4-ifort190-avx512 ls-dyna/12.2.2/1-d8-ifort160-sse2 ls-dyna/15.0.2/mpi-f4-ifort190-sse2 ls-dyna/12.2.2/1-f4-aocc400-avx2 ls-dyna/16.0.0/1-d8-aocc420-avx2 ls-dyna/12.2.2/1-f4-ifort160-sse2 ls-dyna/16.0.0/1-d8-aocc420-avx512 ls-dyna/12.2.2/mpi-d8-aocc400-avx2 ls-dyna/16.0.0/1-d8-ifort190-sse2 ls-dyna/12.2.2/mpi-d8-ifort160-avx2 ls-dyna/16.0.0/1-f4-aocc420-avx2 ls-dyna/12.2.2/mpi-d8-ifort160-sse2 ls-dyna/16.0.0/1-f4-aocc420-avx512 ls-dyna/12.2.2/mpi-f4-aocc400-avx2 ls-dyna/16.0.0/1-f4-ifort190-sse2 ls-dyna/12.2.2/mpi-f4-ifort160-avx2 ls-dyna/16.0.0/mpi-d8-aocc420-avx2 ls-dyna/12.2.2/mpi-f4-ifort160-sse2 ls-dyna/16.0.0/mpi-d8-aocc420-avx512 ls-dyna/13.0.0/1-d8-ifort190 ls-dyna/16.0.0/mpi-d8-ifort190-avx2 ls-dyna/13.0.0/1-f4-ifort190 ls-dyna/16.0.0/mpi-d8-ifort190-avx512 ls-dyna/13.0.0/mpi-d8-ifort190-avx2 ls-dyna/16.0.0/mpi-d8-ifort190-sse2 ls-dyna/13.0.0/mpi-d8-ifort190-sse2 ls-dyna/16.0.0/mpi-f4-aocc420-avx2 ls-dyna/13.0.0/mpi-f4-ifort190-avx2 ls-dyna/16.0.0/mpi-f4-aocc420-avx512 ls-dyna/13.0.0/mpi-f4-ifort190-sse2 ls-dyna/16.0.0/mpi-f4-ifort190-avx2 ls-dyna/13.1.0/mpi-d8-aocc310-avx2 ls-dyna/16.0.0/mpi-f4-ifort190-avx512 ls-dyna/13.1.0/mpi-d8-ifort190-avx2 ls-dyna/16.0.0/mpi-f4-ifort190-sse2 ls-dyna/13.1.0/mpi-d8-ifort190-sse2 ls-dyna/16.1.0/mpi-d8-aocc420-avx2 ls-dyna/13.1.0/mpi-f4-aocc310-avx2 ls-dyna/16.1.0/mpi-d8-aocc420-avx512 ls-dyna/13.1.0/mpi-f4-ifort190-avx2 ls-dyna/16.1.0/mpi-d8-ifort190-avx2 ls-dyna/13.1.0/mpi-f4-ifort190-sse2 ls-dyna/16.1.0/mpi-d8-ifort190-avx512 ls-dyna/13.1.1/mpi-d8-ifort190-avx2 ls-dyna/16.1.0/mpi-d8-ifort190-sse2 ls-dyna/13.1.1/mpi-d8-ifort190-sse2 ls-dyna/16.1.0/mpi-f4-aocc420-avx2 ls-dyna/13.1.1/mpi-f4-ifort190-avx2 ls-dyna/16.1.0/mpi-f4-aocc420-avx512 ls-dyna/13.1.1/mpi-f4-ifort190-sse2 ls-dyna/16.1.0/mpi-f4-ifort190-avx2 ls-dyna/14.0.0/1-d8-aocc310 ls-dyna/16.1.0/mpi-f4-ifort190-avx512 ls-dyna/14.0.0/1-d8-ifort190 ls-dyna/16.1.0/mpi-f4-ifort190-sse2 ls-dyna/14.0.0/1-f4-aocc310 ls-dyna/16.1.1/mpi-d8-aocc420-avx2 ls-dyna/14.0.0/1-f4-ifort190 ls-dyna/16.1.1/mpi-d8-aocc420-avx512 ls-dyna/14.0.0/mpi-d8-aocc310-avx2 ls-dyna/16.1.1/mpi-d8-ifort190-avx2 ls-dyna/14.0.0/mpi-d8-ifort190-avx2 ls-dyna/16.1.1/mpi-d8-ifort190-avx512 ls-dyna/14.0.0/mpi-d8-ifort190-sse2 ls-dyna/16.1.1/mpi-d8-ifort190-sse2 ls-dyna/14.0.0/mpi-f4-ifort190-avx2 ls-dyna/16.1.1/mpi-f4-aocc420-avx2 ls-dyna/14.0.0/mpi-f4-ifort190-sse2 ls-dyna/16.1.1/mpi-f4-aocc420-avx512 ls-dyna/14.1.0/1-d8-aocc400-avx2 ls-dyna/16.1.1/mpi-f4-ifort190-avx2 ls-dyna/14.1.0/1-d8-ifort190-sse2 ls-dyna/16.1.1/mpi-f4-ifort190-avx512 ls-dyna/14.1.0/1-f4-aocc400-avx2 ls-dyna/16.1.1/mpi-f4-ifort190-sse2 ls-dyna/14.1.0/1-f4-ifort190-sse2