Main Page

The ARROW HPC Cluster at TRACC

Operated by Argonne National Laboratory for the U.S. Department of Transportation

Introduction

The ARROW HPC cluster at TRACC is being operated for the Federal Highway Administration (FHWA) and the National Highway Traffic Safety Administration (NTHSA) at the US Department of Transportation. The site operates at Argonne National Laboratory under the Transportation Research and Analysis Computing Center (TRACC). The site is funded by specific US Department of Transportation projects and is not available to the general research community.

TRACC has combined the original hardware from the Phoenix and Zephyr clusters into a single HPC cluster called ARROW. This consolidation allows for the efficient administration of TRACC cluster services. To mitigate the problems with load balancing, the different types of hardware nodes on the ARROW cluster are partitioned into queues with specific characteristics. When new hardware is installed to expand cluster resources, it will be made typically available via a new queue.

ARROW is arranged such that there is a set of four login nodes that are accessed by users using the SSH protocol. The login servers are configured identically so that it doesn't matter which login server a user is connected to (the caveat is that the user should connect to the same login server where he may have created a GUI desktop session). All servers and nodes share a single parallel file system, and an individual home directory is provided for each user to store their data.

Some common software packages provided on the cluster are commercial engineering applications such as StarCCM+, and LS-Dyna. In addition, selected open source software is available, such as OpenFOAM. Specific versions of each software are selected by the user through the "module" command, which is commonly used on HPC clusters. Connections are limited to the SSH protocol at this time, and cyber security is a major aspect of our operation. We will work with you to optimize your remote operating experience. Users will be able to create and use GUI desktops using the XFCE software. These desktops will stay active until the users decides to log out. Otherwise, GUI sessions will stay active and can be reconnected to, similar to the Windows Remote Desktop software. Thus, users can connect and disconnect from these sessions at will, and network interruptions will typically not interfere with your work flow. The PBS job submission framework allows users to use HPC resources in an efficient manner, sharing a fairly complex selection of specialty systems. This includes typical HPC nodes, as well as selected GPU systems for AI/ML applications.

The ARROW HPC cluster consists of about 200 servers and provides a substantial computing facility and research platform for USDOT, especially in the areas of Bridge and River Hydraulics as well as Vehicle Occupant Safety analysis. It has been optimized for HPC engineering applications in Computational Fluid Dynamics and Computational Structural Mechanics.

User Accounts

User accounts are granted to users identified by the U.S. Department of Transportation. The current sponsors for our computing facility are the Federal Highway Administration (FHWA) and the National Highway Traffic Safety Administration (NTHSA) at the US Department of Transportation. You may want to contact Dr. Hubert Ley at Argonne National Laboratory to inquire about the potential support and use of the facility for one of your transportation research projects.

If you are working on a project that is already sponsoring the facility, have your contact person at USDOT request an account for you. We will then send you a link to a form where you can enter all the relevant contact information, and we will then create the account for you. These accounts usually start with "ac.", for example "ac.username", and you will need to remember that when you log in to ARROW. We will communicate all details with you once the application progresses.

Using the Cluster

The following links can be used to find more detailed information on the various aspects of the ARROW cluster's design and operation. This includes information on how to access the cluster remotely, typical applications that can be used to access the cluster, the overall layout of working queues and the job submission procedure, aspects of running specific software applications, information about software compilations, selecting software modules, using graphical applications and user interfaces, information about redundancy and backups, and much more.

• How to Connect
  
• Transferring Files
  
• Job Submission and Monitoring
  

Software Application Documentation

The following links will not work when reading these page with an external browser. They are only available when viewing the pages within the Linux desktop GUI after being logged in.

StarCCM+

• Version 20.04.007-R8
  • StarCCM+ Online User's Guide (English)
  • StarCCM+ Online User's Guide (Chinese)
  • StarCCM+ API Documentation
  • StarCCM+ Java API Documentation
  • StarCCM+ All Manuals and Release Notes

LS-Dyna+

• LS-Dyna Version 16
  • LS-Dyna Theory Manual
  • LS-Dyna User's Manual
  • LS-Dyna User's Manual
  • LS-Dyna User's Manual
  • All Manuals and Release Notes

The links and pages shown below are outdated!

IMPORTANT NOTE: We are updating the wiki pages with the most recent information on services, hardware, and capabilities. The information provided below is outdated and in urgent need for a update. We recommend to disregard the information for now.

General Information

• Becoming_A_User
  

Procedures

• Transferring Files
  
• Setting Up Your Environment
  
• Login Nodes Versus Compute Nodes
  
• Files & Storage
  
• Backups
  
• Compiling Your Code
  
• Running Your Code
  
• Graphical Applications
  

TRACC Cluster Software

• Supported On ARROW
  
• Under Investigation
  

TRACC Policies

• Application Restrictions
  
• Data Retention
  
• Email
  
• Login Nodes