Job Submission and Monitoring: Difference between revisions

From TRACC Wiki
Jump to navigation Jump to search
← Older editNewer edit →
Ley (talk | contribs)
277 edits
Ley (talk | contribs)
277 edits
Line 58: Line 58:
==Job Submission Basics==
==Job Submission Basics==


<I>This chapter is based on the TORQUE syntax for submitting jobs on the cluster. An updated version will be written to explain the syntax of the OpenPBS system, which works very similarly.
<U><I>This chapter is based on the TORQUE syntax for submitting jobs on the cluster. An updated version will be written to explain the syntax of the OpenPBS system, which works very similarly.</I></U>


Sample LS-Dyna job submission script looks like this:
Sample LS-Dyna job submission script looks like this:

Revision as of 05:08, November 20, 2025

Resource Summary View (qsum)

To get started, users can query the overall status of resources on the cluster. The "qsum" script will list all queues and nodes, as well as how many are offline, down, free, or assigned to users. This is a script developed by our team, and may need to be updated if something goes wrong. Please contact us if you experience any problems.

Each queue groups a number of nodes together based on their hardware and software configurations. Nodes can be part of more than one queue, and there are other complex details that we are ignoring here for the purpose of keeping it simple.

Here is a very brief summary of what each of the queues is, and how to use them efficiently:

  • a4000: This is a queue that has three 16-core CPU machines, each of which is furthermore equipped with three A4000 GPUs. That makes a total of 9 A4000 GPUs available to users. Neither the GPUs nor the processors are particularly powerful these days. The machines have 500GB of memory though, which makes for a good platform for experimenting with GPU capabilities.
  • a6000: This is a queue that has only one 64-core CPU machines, and is equipped with four A6000 GPUs. The system can be upgraded to 8 A6000 GPUs if needed. This is a decent GPU machine that can take a solid workload these days. The machine has 750GB of memory, which makes for a good production platform.
  • amd16: This is a queue with many of our older AMD-based 16-core machines, each of which has 30GB of memory. While individual machines are a bit outdated, they are all interconnected with Infiniband and can provide a solid production workload in multi-nodes jobs over MPI without blocking the more current (and thus expensive) systems.
  • epyc1 and epyc2: These are 2 separate queues with slightly different performance characteristics. Each of the groups is interconnected with Infiniband to provide a platform for large and demanding software packages, such as LS-Dyna and StarCCM+. They have between 250GB and 500GB of memory. Because licenses for these software packages are very expensive, they should use these two queues for making optimum use of limited core licenses available to each package.
  1. xeon28: This is a set of intermediate machines with 28 cores and 64GB of memory. They can be used for a variety of purposes, including MPI jobs and single node application software.
  1. virtual: This is a set of nodes without MPI capabilities. They are virtual machines with 32GB each. They can be used for higher demand applications that would interfere with the login nodes, and therefore with other users of these login machines. A user would submit interactive jobs to individual virtual machines and avoid any significant load on login nodes.
$ qsum
=============== a4000 ==========================================================
Queue: "a4000" / nodes: 3 / down: 0 / offline: 0 / busy: 0 / available: 3
      AVAILABLE (3): g001, g002, g003
=============== a6000 ==========================================================
Queue: "a6000" / nodes: 1 / down: 0 / offline: 0 / busy: 0 / available: 1
      AVAILABLE (1): lambda01
=============== amd16 ==========================================================
Queue: "amd16" / nodes: 33 / down: 2 / offline: 0 / busy: 2 / available: 29
           DOWN (2): n017, n030
            ley (2): n001, n002
     AVAILABLE (29): n003, n004, n005, n006, n007, n008, n009, n010, n011, n012
                     n013, n014, n015, n016, n018, n019, n020, n021, n022, n023
                     n024, n025, n026, n027, n028, n029, n031, n032, n039
=============== epyc1 ==========================================================
Queue: "epyc1" / nodes: 1 / down: 0 / offline: 0 / busy: 0 / available: 1
      AVAILABLE (1): a027
=============== epyc2 ==========================================================
Queue: "epyc2" / nodes: 20 / down: 0 / offline: 0 / busy: 5 / available: 15
            ley (2): a030, a031
         msitek (3): a028, a029, a032
     AVAILABLE (15): a033, a034, a035, a036, a037, a038, a039, a040, a041, a042
                     a043, a044, a045, a046, a047
=============== virtual ========================================================
Queue: "virtual" / nodes: 6 / down: 0 / offline: 0 / busy: 0 / available: 6
      AVAILABLE (6): v001, v002, v003, v004, v005, v006
=============== xeon28 =========================================================
Queue: "xeon28" / nodes: 12 / down: 0 / offline: 0 / busy: 0 / available: 12
     AVAILABLE (12): p001, p002, p003, p004, p005, p006, p007, p008, p009, p010
                     p011, p012
================================================================================

Queue Status and Monitoring Jobs(qstat)

Job Submission Basics

This chapter is based on the TORQUE syntax for submitting jobs on the cluster. An updated version will be written to explain the syntax of the OpenPBS system, which works very similarly.

Sample LS-Dyna job submission script looks like this: 

#!/bin/bash
#
#PBS -q extra
#PBS -A ls-dyna
#PBS -l nodes=2:ppn=64
#PBS -l walltime=100::
#PBS -j oe
#PBS -N male_female
#PBS -e log.error
#PBS -o log.output
#PBS -W x=NACCESSPOLICY:SINGLEJOB
#PBS -m bae
module load ls-dyna/MPP/SP/AVX2/12.2.1
cd $PBS_O_WORKDIR
mpirun -machinefile $PBS_NODEFILE ls-dyna i=main.k memory1=300m memory2=100m
module load dynamore/evaluation
DM.plotcprs.lnx -merge

For an interactive session that makes it easier to troubleshoot your jobs during development, you can allocate machines for your job using the following syntax: 

qsub -I -q extra -l nodes=2:ppn=64,walltime=24:00:00

LS-Dyna on the ARROW Cluster

Currently Available LS-Dyna Versions

The following is a list of LS-Dyna versions available on ARROW after the latest reconfiguration of the system. Versions below 12.0.0 are no longer available because they are not compatible with modern operating systems and cannot be made to work correctly.

All versions are loaded using the "module load" command. Versions can be listed with the "module avail ls-dyna" command. To load opne of the modules, use the following syntax: 

module load ls-dyna/14.2.0/mpi-d8-ifort190-avx512

The version string is composed of multiple elements to indicate variants in compilers and compiler options. Use the following guideline to choose an appropriate version to load:

  • "1" or "mpi" indicates whether this is a single node version of LS-Dyna or whether this is a multi-node MPI version. All MPI versions use the IntelMPI 2022 libraries which have been tested thoroughly on ARROW. MPI versions will use the Infiniband Network of ARROW for high-speed and low-latency inter-process communication using RDMA (remote direct memory access).
  • All LS-Dyna versions are available in either floating point or double precision variants. Floating point variants use 4 bytes to represent a value, and double precision variants use 8 bytes. There are pros and cons for choosing one over the other variant. With regards to computational efficiency, both perform nearly the same because all machines are equipped with 64-bit CPUs.
    • "f4" floating point versions
      • Pros: These require significantly less memory to run. Results occupy less disk space, and can be transferred significantly faster into and out of ARROW.
      • Cons: The numerical resolution is limited to 7 significant digits, which is often undesirable when dealing with mathematical operations on small and large numbers at the same time.
    • "r8" double precision versions
      • Pros: The numerical resolution is about twice the number of significant digits compare to "f4", which helps when when dealing with mathematical operations on small and large numbers at the same time.
      • Cons: These require more memory to run. Results occupy more disk space, and it takes longer to transfer data into and out of ARROW.
  • There are two more identifiers to choose from when it comes to the variants of the executables: the specific compiler used to create the executable and the specific processor instruction set required for running the executable.
    • For modern versions of LS-Dyna, two compilers have been used by the developers to create LS-Dyna executables: the Intel Fortran Compiler and the AOCC (AMD Optimizing C/C++ and Fortran) compiler. Both variants of the software are supported on ARROW. This gives users the opportunity to choose an alternate variant of the same LS-Dyna version when running into bugs or crashes.
    • The variants based on the various instruction set extensions (SSE2, AVX2, AVX512, and so on) gives users even more options when choosing an alternate LS-Dyna variant of the same version when running into bugs or crashes. These instruction sets are mostly related to performance gains on specific processors. We have not performed thorough performance tests and cannot recommend specific versions right now.
$ module avail ls-dyna
------------------------------------- /shared/apps/modulefiles -------------------------------------
ls-dyna/12.1.0/1-d8-aocc310          ls-dyna/14.1.0/mpi-d8-aocc400-avx2     
ls-dyna/12.1.0/1-d8-ifort160         ls-dyna/14.1.0/mpi-d8-ifort190-avx2    
ls-dyna/12.1.0/1-f4-aocc310          ls-dyna/14.1.0/mpi-d8-ifort190-avx512  
ls-dyna/12.1.0/1-f4-ifort160         ls-dyna/14.1.0/mpi-d8-ifort190-sse2    
ls-dyna/12.1.0/mpi-d8-aocc310-avx2   ls-dyna/14.1.0/mpi-f4-aocc400-avx2     
ls-dyna/12.1.0/mpi-d8-ifort160-avx2  ls-dyna/14.1.0/mpi-f4-ifort190-avx2    
ls-dyna/12.1.0/mpi-d8-ifort160-sse2  ls-dyna/14.1.0/mpi-f4-ifort190-avx512  
ls-dyna/12.1.0/mpi-f4-aocc310-avx2   ls-dyna/14.1.0/mpi-f4-ifort190-sse2    
ls-dyna/12.1.0/mpi-f4-ifort160-avx2  ls-dyna/14.2.0/1-d8-aocc400-avx2       
ls-dyna/12.1.0/mpi-f4-ifort160-sse2  ls-dyna/14.2.0/1-d8-ifort190-sse2      
ls-dyna/12.2.0/1-d8-aocc400          ls-dyna/14.2.0/1-f4-aocc400-avx2       
ls-dyna/12.2.0/1-d8-ifort160         ls-dyna/14.2.0/1-f4-ifort190-sse2      
ls-dyna/12.2.0/1-f4-aocc400          ls-dyna/14.2.0/mpi-d8-aocc400-avx2     
ls-dyna/12.2.0/1-f4-ifort160         ls-dyna/14.2.0/mpi-d8-ifort190-avx2    
ls-dyna/12.2.0/mpi-d8-aocc400-avx2   ls-dyna/14.2.0/mpi-d8-ifort190-avx512  
ls-dyna/12.2.0/mpi-d8-ifort160-avx2  ls-dyna/14.2.0/mpi-d8-ifort190-sse2    
ls-dyna/12.2.0/mpi-d8-ifort160-sse2  ls-dyna/14.2.0/mpi-f4-aocc400-avx2     
ls-dyna/12.2.0/mpi-f4-aocc400-avx2   ls-dyna/14.2.0/mpi-f4-ifort190-avx2    
ls-dyna/12.2.0/mpi-f4-ifort160-avx2  ls-dyna/14.2.0/mpi-f4-ifort190-avx512  
ls-dyna/12.2.0/mpi-f4-ifort160-sse2  ls-dyna/14.2.0/mpi-f4-ifort190-sse2    
ls-dyna/12.2.1/1-d8-aocc400-avx2     ls-dyna/15.0.2/1-d8-aocc400-avx2       
ls-dyna/12.2.1/1-d8-ifort160-sse2    ls-dyna/15.0.2/1-d8-ifort190-sse2      
ls-dyna/12.2.1/1-f4-aocc400-avx2     ls-dyna/15.0.2/1-f4-aocc400-avx2       
ls-dyna/12.2.1/1-f4-ifort160-sse2    ls-dyna/15.0.2/1-f4-ifort190-sse2      
ls-dyna/12.2.1/mpi-d8-aocc400-avx2   ls-dyna/15.0.2/mpi-d8-aocc400-avx2     
ls-dyna/12.2.1/mpi-d8-ifort160-avx2  ls-dyna/15.0.2/mpi-d8-ifort190-avx2    
ls-dyna/12.2.1/mpi-d8-ifort160-sse2  ls-dyna/15.0.2/mpi-d8-ifort190-avx512  
ls-dyna/12.2.1/mpi-f4-aocc400-avx2   ls-dyna/15.0.2/mpi-d8-ifort190-sse2    
ls-dyna/12.2.1/mpi-f4-ifort160-avx2  ls-dyna/15.0.2/mpi-f4-aocc400-avx2     
ls-dyna/12.2.1/mpi-f4-ifort160-sse2  ls-dyna/15.0.2/mpi-f4-ifort190-avx2    
ls-dyna/12.2.2/1-d8-aocc400-avx2     ls-dyna/15.0.2/mpi-f4-ifort190-avx512  
ls-dyna/12.2.2/1-d8-ifort160-sse2    ls-dyna/15.0.2/mpi-f4-ifort190-sse2    
ls-dyna/12.2.2/1-f4-aocc400-avx2     ls-dyna/16.0.0/1-d8-aocc420-avx2       
ls-dyna/12.2.2/1-f4-ifort160-sse2    ls-dyna/16.0.0/1-d8-aocc420-avx512     
ls-dyna/12.2.2/mpi-d8-aocc400-avx2   ls-dyna/16.0.0/1-d8-ifort190-sse2      
ls-dyna/12.2.2/mpi-d8-ifort160-avx2  ls-dyna/16.0.0/1-f4-aocc420-avx2       
ls-dyna/12.2.2/mpi-d8-ifort160-sse2  ls-dyna/16.0.0/1-f4-aocc420-avx512     
ls-dyna/12.2.2/mpi-f4-aocc400-avx2   ls-dyna/16.0.0/1-f4-ifort190-sse2      
ls-dyna/12.2.2/mpi-f4-ifort160-avx2  ls-dyna/16.0.0/mpi-d8-aocc420-avx2     
ls-dyna/12.2.2/mpi-f4-ifort160-sse2  ls-dyna/16.0.0/mpi-d8-aocc420-avx512   
ls-dyna/13.0.0/1-d8-ifort190         ls-dyna/16.0.0/mpi-d8-ifort190-avx2    
ls-dyna/13.0.0/1-f4-ifort190         ls-dyna/16.0.0/mpi-d8-ifort190-avx512  
ls-dyna/13.0.0/mpi-d8-ifort190-avx2  ls-dyna/16.0.0/mpi-d8-ifort190-sse2    
ls-dyna/13.0.0/mpi-d8-ifort190-sse2  ls-dyna/16.0.0/mpi-f4-aocc420-avx2     
ls-dyna/13.0.0/mpi-f4-ifort190-avx2  ls-dyna/16.0.0/mpi-f4-aocc420-avx512   
ls-dyna/13.0.0/mpi-f4-ifort190-sse2  ls-dyna/16.0.0/mpi-f4-ifort190-avx2    
ls-dyna/13.1.0/mpi-d8-aocc310-avx2   ls-dyna/16.0.0/mpi-f4-ifort190-avx512  
ls-dyna/13.1.0/mpi-d8-ifort190-avx2  ls-dyna/16.0.0/mpi-f4-ifort190-sse2    
ls-dyna/13.1.0/mpi-d8-ifort190-sse2  ls-dyna/16.1.0/mpi-d8-aocc420-avx2     
ls-dyna/13.1.0/mpi-f4-aocc310-avx2   ls-dyna/16.1.0/mpi-d8-aocc420-avx512   
ls-dyna/13.1.0/mpi-f4-ifort190-avx2  ls-dyna/16.1.0/mpi-d8-ifort190-avx2    
ls-dyna/13.1.0/mpi-f4-ifort190-sse2  ls-dyna/16.1.0/mpi-d8-ifort190-avx512  
ls-dyna/13.1.1/mpi-d8-ifort190-avx2  ls-dyna/16.1.0/mpi-d8-ifort190-sse2    
ls-dyna/13.1.1/mpi-d8-ifort190-sse2  ls-dyna/16.1.0/mpi-f4-aocc420-avx2     
ls-dyna/13.1.1/mpi-f4-ifort190-avx2  ls-dyna/16.1.0/mpi-f4-aocc420-avx512   
ls-dyna/13.1.1/mpi-f4-ifort190-sse2  ls-dyna/16.1.0/mpi-f4-ifort190-avx2    
ls-dyna/14.0.0/1-d8-aocc310          ls-dyna/16.1.0/mpi-f4-ifort190-avx512  
ls-dyna/14.0.0/1-d8-ifort190         ls-dyna/16.1.0/mpi-f4-ifort190-sse2    
ls-dyna/14.0.0/1-f4-aocc310          ls-dyna/16.1.1/mpi-d8-aocc420-avx2     
ls-dyna/14.0.0/1-f4-ifort190         ls-dyna/16.1.1/mpi-d8-aocc420-avx512   
ls-dyna/14.0.0/mpi-d8-aocc310-avx2   ls-dyna/16.1.1/mpi-d8-ifort190-avx2    
ls-dyna/14.0.0/mpi-d8-ifort190-avx2  ls-dyna/16.1.1/mpi-d8-ifort190-avx512  
ls-dyna/14.0.0/mpi-d8-ifort190-sse2  ls-dyna/16.1.1/mpi-d8-ifort190-sse2    
ls-dyna/14.0.0/mpi-f4-ifort190-avx2  ls-dyna/16.1.1/mpi-f4-aocc420-avx2     
ls-dyna/14.0.0/mpi-f4-ifort190-sse2  ls-dyna/16.1.1/mpi-f4-aocc420-avx512   
ls-dyna/14.1.0/1-d8-aocc400-avx2     ls-dyna/16.1.1/mpi-f4-ifort190-avx2    
ls-dyna/14.1.0/1-d8-ifort190-sse2    ls-dyna/16.1.1/mpi-f4-ifort190-avx512  
ls-dyna/14.1.0/1-f4-aocc400-avx2     ls-dyna/16.1.1/mpi-f4-ifort190-sse2    
ls-dyna/14.1.0/1-f4-ifort190-sse2

LSTC Tools: LS-OPT and LS-PREPOST

Dynamore Software

Vendor License File Installation

StarCCM+ on the ARROW Cluster

Currently Available StarCCM+ Versions

OpenFOAM on the ARROW Cluster

Currently Available OpenFOAM Versions

Additional Software Applications and Libraries

Loadable GCC Compiler Versions

MPI Libraries and Runtimes

MatLab Runtimes

Retrieved from "https://wiki.anl.gov/wiki_tracc/index.php?title=Job_Submission_and_Monitoring&oldid=2337"