Job Submission and Monitoring

Resource Summary View (qsum)

To get started, users can query the overall status of resources on the cluster. The "qsum" script will list all queues and nodes, as well as how many are offline, down, free, or assigned to users. This is a script developed by our team, and may need to be updated if something goes wrong. Please contact us if you experience any problems.

Each queue groups a number of nodes together based on their hardware and software configurations. Nodes can be part of more than one queue, and there are other complex details that we are ignoring here for the purpose of keeping it simple.

Here is a very brief summary of what each of the queues is, and how to use them efficiently:

• a4000: This is a queue that has three 16-core CPU machines, each of which is furthermore equipped with three A4000 GPUs. That makes a total of 9 A4000 GPUs available to users. Neither the GPUs nor the processors are particularly powerful these days. The machines have 512GB of memory though, which makes for a good platform for experimenting with GPU capabilities.

$ qsum
=============== a4000 ==========================================================
Queue: "a4000" / nodes: 3 / down: 0 / offline: 0 / busy: 0 / available: 3
      AVAILABLE (3): g001, g002, g003
=============== a6000 ==========================================================
Queue: "a6000" / nodes: 1 / down: 0 / offline: 0 / busy: 0 / available: 1
      AVAILABLE (1): lambda01
=============== amd16 ==========================================================
Queue: "amd16" / nodes: 33 / down: 2 / offline: 0 / busy: 2 / available: 29
           DOWN (2): n017, n030
            ley (2): n001, n002
     AVAILABLE (29): n003, n004, n005, n006, n007, n008, n009, n010, n011, n012
                     n013, n014, n015, n016, n018, n019, n020, n021, n022, n023
                     n024, n025, n026, n027, n028, n029, n031, n032, n039
=============== epyc1 ==========================================================
Queue: "epyc1" / nodes: 1 / down: 0 / offline: 0 / busy: 0 / available: 1
      AVAILABLE (1): a027
=============== epyc2 ==========================================================
Queue: "epyc2" / nodes: 20 / down: 0 / offline: 0 / busy: 5 / available: 15
            ley (2): a030, a031
         msitek (3): a028, a029, a032
     AVAILABLE (15): a033, a034, a035, a036, a037, a038, a039, a040, a041, a042
                     a043, a044, a045, a046, a047
=============== virtual ========================================================
Queue: "virtual" / nodes: 6 / down: 0 / offline: 0 / busy: 0 / available: 6
      AVAILABLE (6): v001, v002, v003, v004, v005, v006
=============== xeon28 =========================================================
Queue: "xeon28" / nodes: 12 / down: 0 / offline: 0 / busy: 0 / available: 12
     AVAILABLE (12): p001, p002, p003, p004, p005, p006, p007, p008, p009, p010
                     p011, p012
================================================================================

Jobs Submission Basics

Sample LS-Dyna job submission script looks like this:

#!/bin/bash
#
#PBS -q extra
#PBS -A ls-dyna
#PBS -l nodes=2:ppn=64
#PBS -l walltime=100::
#PBS -j oe
#PBS -N male_female
#PBS -e log.error
#PBS -o log.output
#PBS -W x=NACCESSPOLICY:SINGLEJOB
#PBS -m bae
module load ls-dyna/MPP/SP/AVX2/12.2.1
cd $PBS_O_WORKDIR
mpirun -machinefile $PBS_NODEFILE ls-dyna i=main.k memory1=300m memory2=100m
module load dynamore/evaluation
DM.plotcprs.lnx -merge

For an interactive session that makes it easier to troubleshoot your jobs during development, you can allocate machines for your job using the following syntax:

qsub -I -q extra -l nodes=2:ppn=64,walltime=24:00:00

LS-Dyna on the ARROW Cluster

Currently Available LS-Dyna Versions

The following is a list of LS-Dyna versions available on ARROW after the latest reconfiguration of the system. Versions below 12.0.0 are no longer available because they are not compatible with modern operating systems and cannot be made to work correctly.

All versions are loaded using the "module load" command. Versions can be listed with the "module avail ls-dyna" command. To load opne of the modules, use the following syntax:

module load ls-dyna/14.2.0/mpi-d8-ifort190-avx512

The version string is composed of multiple elements to indicate variants in compilers and compiler options. Use the following guideline to choose an appropriate version to load:

• "1" or "mpi" indicates whether this is a single node version of LS-Dyna or whether this is a multi-node MPI version. All MPI versions use the IntelMPI 2022 libraries which have been tested thoroughly on ARROW. MPI versions will use the Infiniband Network of ARROW for high-speed and low-latency inter-process communication using RDMA (remote direct memory access).
• All LS-Dyna versions are available in either floating point or double precision variants. Floating point variants use 4 bytes to represent a value, and double precision variants use 8 bytes. There are pros and cons for choosing one over the other variant. With regards to computational efficiency, both perform nearly the same because all machines are equipped with 64-bit CPUs.
  • "f4" floating point versions
    • Pros: These require significantly less memory to run. Results occupy less disk space, and can be transferred significantly faster into and out of ARROW.
    • Cons: The numerical resolution is limited to 7 significant digits, which is often undesirable when dealing with mathematical operations on small and large numbers at the same time.
  • "r8" double precision versions
    • Pros: The numerical resolution is about twice the number of significant digits compare to "f4", which helps when when dealing with mathematical operations on small and large numbers at the same time.
    • Cons: These require more memory to run. Results occupy more disk space, and it takes longer to transfer data into and out of ARROW.
• There are two more identifiers to choose from when it comes to the variants of the executables: the specific compiler used to create the executable and the specific processor instruction set required for running the executable.
  • For modern versions of LS-Dyna, two compilers have been used by the developers to create LS-Dyna executables: the Intel Fortran Compiler and the AOCC (AMD Optimizing C/C++ and Fortran) compiler. Both variants of the software are supported on ARROW. This gives users the opportunity to choose an alternate variant of the same LS-Dyna version when running into bugs or crashes.
  • The variants based on the various instruction set extensions (SSE2, AVX2, AVX512, and so on) gives users even more options when choosing an alternate LS-Dyna variant of the same version when running into bugs or crashes. These instruction sets are mostly related to performance gains on specific processors. We have not performed thorough performance tests and cannot recommend specific versions right now.

$ module avail ls-dyna
------------------------------------- /shared/apps/modulefiles -------------------------------------
ls-dyna/12.1.0/1-d8-aocc310          ls-dyna/14.1.0/mpi-d8-aocc400-avx2     
ls-dyna/12.1.0/1-d8-ifort160         ls-dyna/14.1.0/mpi-d8-ifort190-avx2    
ls-dyna/12.1.0/1-f4-aocc310          ls-dyna/14.1.0/mpi-d8-ifort190-avx512  
ls-dyna/12.1.0/1-f4-ifort160         ls-dyna/14.1.0/mpi-d8-ifort190-sse2    
ls-dyna/12.1.0/mpi-d8-aocc310-avx2   ls-dyna/14.1.0/mpi-f4-aocc400-avx2     
ls-dyna/12.1.0/mpi-d8-ifort160-avx2  ls-dyna/14.1.0/mpi-f4-ifort190-avx2    
ls-dyna/12.1.0/mpi-d8-ifort160-sse2  ls-dyna/14.1.0/mpi-f4-ifort190-avx512  
ls-dyna/12.1.0/mpi-f4-aocc310-avx2   ls-dyna/14.1.0/mpi-f4-ifort190-sse2    
ls-dyna/12.1.0/mpi-f4-ifort160-avx2  ls-dyna/14.2.0/1-d8-aocc400-avx2       
ls-dyna/12.1.0/mpi-f4-ifort160-sse2  ls-dyna/14.2.0/1-d8-ifort190-sse2      
ls-dyna/12.2.0/1-d8-aocc400          ls-dyna/14.2.0/1-f4-aocc400-avx2       
ls-dyna/12.2.0/1-d8-ifort160         ls-dyna/14.2.0/1-f4-ifort190-sse2      
ls-dyna/12.2.0/1-f4-aocc400          ls-dyna/14.2.0/mpi-d8-aocc400-avx2     
ls-dyna/12.2.0/1-f4-ifort160         ls-dyna/14.2.0/mpi-d8-ifort190-avx2    
ls-dyna/12.2.0/mpi-d8-aocc400-avx2   ls-dyna/14.2.0/mpi-d8-ifort190-avx512  
ls-dyna/12.2.0/mpi-d8-ifort160-avx2  ls-dyna/14.2.0/mpi-d8-ifort190-sse2    
ls-dyna/12.2.0/mpi-d8-ifort160-sse2  ls-dyna/14.2.0/mpi-f4-aocc400-avx2     
ls-dyna/12.2.0/mpi-f4-aocc400-avx2   ls-dyna/14.2.0/mpi-f4-ifort190-avx2    
ls-dyna/12.2.0/mpi-f4-ifort160-avx2  ls-dyna/14.2.0/mpi-f4-ifort190-avx512  
ls-dyna/12.2.0/mpi-f4-ifort160-sse2  ls-dyna/14.2.0/mpi-f4-ifort190-sse2    
ls-dyna/12.2.1/1-d8-aocc400-avx2     ls-dyna/15.0.2/1-d8-aocc400-avx2       
ls-dyna/12.2.1/1-d8-ifort160-sse2    ls-dyna/15.0.2/1-d8-ifort190-sse2      
ls-dyna/12.2.1/1-f4-aocc400-avx2     ls-dyna/15.0.2/1-f4-aocc400-avx2       
ls-dyna/12.2.1/1-f4-ifort160-sse2    ls-dyna/15.0.2/1-f4-ifort190-sse2      
ls-dyna/12.2.1/mpi-d8-aocc400-avx2   ls-dyna/15.0.2/mpi-d8-aocc400-avx2     
ls-dyna/12.2.1/mpi-d8-ifort160-avx2  ls-dyna/15.0.2/mpi-d8-ifort190-avx2    
ls-dyna/12.2.1/mpi-d8-ifort160-sse2  ls-dyna/15.0.2/mpi-d8-ifort190-avx512  
ls-dyna/12.2.1/mpi-f4-aocc400-avx2   ls-dyna/15.0.2/mpi-d8-ifort190-sse2    
ls-dyna/12.2.1/mpi-f4-ifort160-avx2  ls-dyna/15.0.2/mpi-f4-aocc400-avx2     
ls-dyna/12.2.1/mpi-f4-ifort160-sse2  ls-dyna/15.0.2/mpi-f4-ifort190-avx2    
ls-dyna/12.2.2/1-d8-aocc400-avx2     ls-dyna/15.0.2/mpi-f4-ifort190-avx512  
ls-dyna/12.2.2/1-d8-ifort160-sse2    ls-dyna/15.0.2/mpi-f4-ifort190-sse2    
ls-dyna/12.2.2/1-f4-aocc400-avx2     ls-dyna/16.0.0/1-d8-aocc420-avx2       
ls-dyna/12.2.2/1-f4-ifort160-sse2    ls-dyna/16.0.0/1-d8-aocc420-avx512     
ls-dyna/12.2.2/mpi-d8-aocc400-avx2   ls-dyna/16.0.0/1-d8-ifort190-sse2      
ls-dyna/12.2.2/mpi-d8-ifort160-avx2  ls-dyna/16.0.0/1-f4-aocc420-avx2       
ls-dyna/12.2.2/mpi-d8-ifort160-sse2  ls-dyna/16.0.0/1-f4-aocc420-avx512     
ls-dyna/12.2.2/mpi-f4-aocc400-avx2   ls-dyna/16.0.0/1-f4-ifort190-sse2      
ls-dyna/12.2.2/mpi-f4-ifort160-avx2  ls-dyna/16.0.0/mpi-d8-aocc420-avx2     
ls-dyna/12.2.2/mpi-f4-ifort160-sse2  ls-dyna/16.0.0/mpi-d8-aocc420-avx512   
ls-dyna/13.0.0/1-d8-ifort190         ls-dyna/16.0.0/mpi-d8-ifort190-avx2    
ls-dyna/13.0.0/1-f4-ifort190         ls-dyna/16.0.0/mpi-d8-ifort190-avx512  
ls-dyna/13.0.0/mpi-d8-ifort190-avx2  ls-dyna/16.0.0/mpi-d8-ifort190-sse2    
ls-dyna/13.0.0/mpi-d8-ifort190-sse2  ls-dyna/16.0.0/mpi-f4-aocc420-avx2     
ls-dyna/13.0.0/mpi-f4-ifort190-avx2  ls-dyna/16.0.0/mpi-f4-aocc420-avx512   
ls-dyna/13.0.0/mpi-f4-ifort190-sse2  ls-dyna/16.0.0/mpi-f4-ifort190-avx2    
ls-dyna/13.1.0/mpi-d8-aocc310-avx2   ls-dyna/16.0.0/mpi-f4-ifort190-avx512  
ls-dyna/13.1.0/mpi-d8-ifort190-avx2  ls-dyna/16.0.0/mpi-f4-ifort190-sse2    
ls-dyna/13.1.0/mpi-d8-ifort190-sse2  ls-dyna/16.1.0/mpi-d8-aocc420-avx2     
ls-dyna/13.1.0/mpi-f4-aocc310-avx2   ls-dyna/16.1.0/mpi-d8-aocc420-avx512   
ls-dyna/13.1.0/mpi-f4-ifort190-avx2  ls-dyna/16.1.0/mpi-d8-ifort190-avx2    
ls-dyna/13.1.0/mpi-f4-ifort190-sse2  ls-dyna/16.1.0/mpi-d8-ifort190-avx512  
ls-dyna/13.1.1/mpi-d8-ifort190-avx2  ls-dyna/16.1.0/mpi-d8-ifort190-sse2    
ls-dyna/13.1.1/mpi-d8-ifort190-sse2  ls-dyna/16.1.0/mpi-f4-aocc420-avx2     
ls-dyna/13.1.1/mpi-f4-ifort190-avx2  ls-dyna/16.1.0/mpi-f4-aocc420-avx512   
ls-dyna/13.1.1/mpi-f4-ifort190-sse2  ls-dyna/16.1.0/mpi-f4-ifort190-avx2    
ls-dyna/14.0.0/1-d8-aocc310          ls-dyna/16.1.0/mpi-f4-ifort190-avx512  
ls-dyna/14.0.0/1-d8-ifort190         ls-dyna/16.1.0/mpi-f4-ifort190-sse2    
ls-dyna/14.0.0/1-f4-aocc310          ls-dyna/16.1.1/mpi-d8-aocc420-avx2     
ls-dyna/14.0.0/1-f4-ifort190         ls-dyna/16.1.1/mpi-d8-aocc420-avx512   
ls-dyna/14.0.0/mpi-d8-aocc310-avx2   ls-dyna/16.1.1/mpi-d8-ifort190-avx2    
ls-dyna/14.0.0/mpi-d8-ifort190-avx2  ls-dyna/16.1.1/mpi-d8-ifort190-avx512  
ls-dyna/14.0.0/mpi-d8-ifort190-sse2  ls-dyna/16.1.1/mpi-d8-ifort190-sse2    
ls-dyna/14.0.0/mpi-f4-ifort190-avx2  ls-dyna/16.1.1/mpi-f4-aocc420-avx2     
ls-dyna/14.0.0/mpi-f4-ifort190-sse2  ls-dyna/16.1.1/mpi-f4-aocc420-avx512   
ls-dyna/14.1.0/1-d8-aocc400-avx2     ls-dyna/16.1.1/mpi-f4-ifort190-avx2    
ls-dyna/14.1.0/1-d8-ifort190-sse2    ls-dyna/16.1.1/mpi-f4-ifort190-avx512  
ls-dyna/14.1.0/1-f4-aocc400-avx2     ls-dyna/16.1.1/mpi-f4-ifort190-sse2    
ls-dyna/14.1.0/1-f4-ifort190-sse2