<?xml version="1.0"?>
<feed xmlns="http://www.w3.org/2005/Atom" xml:lang="en">
	<id>https://wiki.anl.gov/wiki_tracc/api.php?action=feedcontributions&amp;feedformat=atom&amp;user=Ley</id>
	<title>TRACC Wiki - User contributions [en]</title>
	<link rel="self" type="application/atom+xml" href="https://wiki.anl.gov/wiki_tracc/api.php?action=feedcontributions&amp;feedformat=atom&amp;user=Ley"/>
	<link rel="alternate" type="text/html" href="https://wiki.anl.gov/tracc/Special:Contributions/Ley"/>
	<updated>2026-06-04T00:13:26Z</updated>
	<subtitle>User contributions</subtitle>
	<generator>MediaWiki 1.43.8</generator>
	<entry>
		<id>https://wiki.anl.gov/wiki_tracc/index.php?title=Private/Public_Key_Authentication&amp;diff=2524</id>
		<title>Private/Public Key Authentication</title>
		<link rel="alternate" type="text/html" href="https://wiki.anl.gov/wiki_tracc/index.php?title=Private/Public_Key_Authentication&amp;diff=2524"/>
		<updated>2026-04-15T14:51:02Z</updated>

		<summary type="html">&lt;p&gt;Ley: /* Configuring WinSCP */&lt;/p&gt;
&lt;hr /&gt;
&lt;div&gt;==Creating a Private/Public Key Pair==&lt;br /&gt;
&lt;br /&gt;
If you have never use public key authentication before, you will have to start by creating a key pair. Once the key pair is created, you need to keep the private key in a safe location. Never upload the private key, and never send it by Email. Whoever has your private key can use it to try logging in to ARROW. So, treat the private key as a secret at all times.&lt;br /&gt;
&lt;br /&gt;
The easiest way to prepare a key pair is the use of &amp;quot;puttygen&amp;quot;, part of the &amp;quot;putty&amp;quot; software on Windows. Pressing the Windows Key together with R will open a window where you enter &amp;quot;puttygen&amp;quot;. Once you press &amp;quot;Ok&amp;quot;, the software opens up:&lt;br /&gt;
&lt;br /&gt;
[[File:Puttygen1.png]]&lt;br /&gt;
&lt;br /&gt;
We can go with the defaults and just click on the &amp;quot;Generate&amp;quot; button. Some randomness depends on you moving the mouse around on the screen. Once enough random data is gathered, the private and public key will be generated.&lt;br /&gt;
&lt;br /&gt;
[[File:Puttygen2.png]]&lt;br /&gt;
&lt;br /&gt;
Once the keys have been generated, you need to store the private key in a easy to find location on your local file system. I recommend creating a folder &amp;quot;Keys&amp;quot; within your &amp;quot;Documents&amp;quot; folder, but any other location is fine as well. The idea is that you may easily forget where the key is stored when you need it a later point in time. &lt;br /&gt;
&lt;br /&gt;
[[File:Puttygen3.png]]&lt;br /&gt;
&lt;br /&gt;
Click on &amp;quot;Save private key&amp;quot;, which will create the following dialog box for you:&lt;br /&gt;
&lt;br /&gt;
[[File:Puttygen4.png]]&lt;br /&gt;
&lt;br /&gt;
In an ideal world, assigning a password for your private key would be best, but that also means that you may have to enter that password every time you use the key, which can be a problem. Not assigning a password is one more reason to never risk losing this private key file.&lt;br /&gt;
&lt;br /&gt;
Navigate to the &amp;quot;Keys&amp;quot; folder in the &amp;quot;Documents: folder, and give the key a meaningful name. Something that tells you what it is used for, like &amp;quot;arrow-private&amp;quot;. The extension ppk is automatically added when the key is saved.&lt;br /&gt;
&lt;br /&gt;
Regarding the public key: It is not necessary to save the public key. When loading a private key, the public key is automatically displayed again at the top of the &amp;quot;puttygen&amp;quot; window. It looks somewhat like this:&lt;br /&gt;
&lt;br /&gt;
==Uploading the public key to one of the ARROW login nodes==&lt;br /&gt;
&lt;br /&gt;
The &amp;quot;puttygen&amp;quot; window still shows the public key at the top.&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
ssh-rsa AAAAB3NzaC1yc2EAfAADAQABAAABAQDOHgbwUYAAo+BMpvskvLw6eTqqs8OVUMSbAIobbC2/518PpJ/b&lt;br /&gt;
jW0n9NM4jlNJlEpuT79ZSv5RwSW8rP82qtcLdSEisxhXkyRLLjbzRWQSGxlVUI+K6MfZeFRYD+nqEH4Q1nSvxWJM&lt;br /&gt;
FigswLuWsygcJnGxv0FBGBP7pUrK5pz6AEv5Blf0MCuy0bvuIGhOYEMO4JeId6WYrAwJmZ3mjHqVwO3uQtrYP1Gi&lt;br /&gt;
71A+yk4x2hZepwW5fHiTbeCU/kDycqWPsXOI3e+eBIDBHmaaD7uOkIWT/GnSBTvYr4NkF2xVxnmCsQ5eks13QdY6&lt;br /&gt;
MX1/gjwLl3nOPl6HBLc+KiHPLlMB rsa-key-20260325&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
You can use your mouse to highlight the entire string (starting at ssh-rsa and ending with the date) and copy it into your copy and paste buffer (this is actually a single line altogether). But first we want to do the following:&lt;br /&gt;
&lt;br /&gt;
* create a terminal connection to one of the login nodes (putty, or even ThinLinc)&lt;br /&gt;
* navigate to the folder &amp;quot;~/.ssh&amp;quot;&lt;br /&gt;
* edit or create a file called &amp;quot;authorized_keys&amp;quot;&lt;br /&gt;
* paste the key from puttygen into this file (it should be a single line of text)&lt;br /&gt;
* save the file&lt;br /&gt;
&lt;br /&gt;
==Using the private key to log in==&lt;br /&gt;
&lt;br /&gt;
The procedure is different for the various pieces of software that can be used to connect to the cluster.&lt;br /&gt;
&lt;br /&gt;
==Configuring PuTTY Connections==&lt;br /&gt;
&lt;br /&gt;
When creating or editing a PuTTY connection, connect your private key as shown here:&lt;br /&gt;
&lt;br /&gt;
[[File:Putty.png]]&lt;br /&gt;
&lt;br /&gt;
Once you save your connection, the key is stored for all future connections to this machine.&lt;br /&gt;
&lt;br /&gt;
==Configuring WinSCP==&lt;br /&gt;
&lt;br /&gt;
When creating or editing a WinSCP connection, connect your private key as shown here. Start with the WinSCP page where you choose your connection, then click on &amp;quot;Advanced&amp;quot;, and configure the connection as shown:&lt;br /&gt;
&lt;br /&gt;
[[File:Winscp.png]]&lt;br /&gt;
&lt;br /&gt;
Once you save your connection, the key is stored for all future connections to this machine.&lt;br /&gt;
&lt;br /&gt;
==Configuring FileZilla==&lt;br /&gt;
&lt;br /&gt;
When creating or editing a FileZilla connection (see Edit/Settings), connect your private key as shown here:&lt;br /&gt;
&lt;br /&gt;
[[File:Filezilla.png]]&lt;br /&gt;
&lt;br /&gt;
Once you save your connection, the key is stored for all future connections to this machine.&lt;/div&gt;</summary>
		<author><name>Ley</name></author>
	</entry>
	<entry>
		<id>https://wiki.anl.gov/wiki_tracc/index.php?title=Private/Public_Key_Authentication&amp;diff=2523</id>
		<title>Private/Public Key Authentication</title>
		<link rel="alternate" type="text/html" href="https://wiki.anl.gov/wiki_tracc/index.php?title=Private/Public_Key_Authentication&amp;diff=2523"/>
		<updated>2026-03-25T21:09:29Z</updated>

		<summary type="html">&lt;p&gt;Ley: /* Configuring FileZilla */&lt;/p&gt;
&lt;hr /&gt;
&lt;div&gt;==Creating a Private/Public Key Pair==&lt;br /&gt;
&lt;br /&gt;
If you have never use public key authentication before, you will have to start by creating a key pair. Once the key pair is created, you need to keep the private key in a safe location. Never upload the private key, and never send it by Email. Whoever has your private key can use it to try logging in to ARROW. So, treat the private key as a secret at all times.&lt;br /&gt;
&lt;br /&gt;
The easiest way to prepare a key pair is the use of &amp;quot;puttygen&amp;quot;, part of the &amp;quot;putty&amp;quot; software on Windows. Pressing the Windows Key together with R will open a window where you enter &amp;quot;puttygen&amp;quot;. Once you press &amp;quot;Ok&amp;quot;, the software opens up:&lt;br /&gt;
&lt;br /&gt;
[[File:Puttygen1.png]]&lt;br /&gt;
&lt;br /&gt;
We can go with the defaults and just click on the &amp;quot;Generate&amp;quot; button. Some randomness depends on you moving the mouse around on the screen. Once enough random data is gathered, the private and public key will be generated.&lt;br /&gt;
&lt;br /&gt;
[[File:Puttygen2.png]]&lt;br /&gt;
&lt;br /&gt;
Once the keys have been generated, you need to store the private key in a easy to find location on your local file system. I recommend creating a folder &amp;quot;Keys&amp;quot; within your &amp;quot;Documents&amp;quot; folder, but any other location is fine as well. The idea is that you may easily forget where the key is stored when you need it a later point in time. &lt;br /&gt;
&lt;br /&gt;
[[File:Puttygen3.png]]&lt;br /&gt;
&lt;br /&gt;
Click on &amp;quot;Save private key&amp;quot;, which will create the following dialog box for you:&lt;br /&gt;
&lt;br /&gt;
[[File:Puttygen4.png]]&lt;br /&gt;
&lt;br /&gt;
In an ideal world, assigning a password for your private key would be best, but that also means that you may have to enter that password every time you use the key, which can be a problem. Not assigning a password is one more reason to never risk losing this private key file.&lt;br /&gt;
&lt;br /&gt;
Navigate to the &amp;quot;Keys&amp;quot; folder in the &amp;quot;Documents: folder, and give the key a meaningful name. Something that tells you what it is used for, like &amp;quot;arrow-private&amp;quot;. The extension ppk is automatically added when the key is saved.&lt;br /&gt;
&lt;br /&gt;
Regarding the public key: It is not necessary to save the public key. When loading a private key, the public key is automatically displayed again at the top of the &amp;quot;puttygen&amp;quot; window. It looks somewhat like this:&lt;br /&gt;
&lt;br /&gt;
==Uploading the public key to one of the ARROW login nodes==&lt;br /&gt;
&lt;br /&gt;
The &amp;quot;puttygen&amp;quot; window still shows the public key at the top.&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
ssh-rsa AAAAB3NzaC1yc2EAfAADAQABAAABAQDOHgbwUYAAo+BMpvskvLw6eTqqs8OVUMSbAIobbC2/518PpJ/b&lt;br /&gt;
jW0n9NM4jlNJlEpuT79ZSv5RwSW8rP82qtcLdSEisxhXkyRLLjbzRWQSGxlVUI+K6MfZeFRYD+nqEH4Q1nSvxWJM&lt;br /&gt;
FigswLuWsygcJnGxv0FBGBP7pUrK5pz6AEv5Blf0MCuy0bvuIGhOYEMO4JeId6WYrAwJmZ3mjHqVwO3uQtrYP1Gi&lt;br /&gt;
71A+yk4x2hZepwW5fHiTbeCU/kDycqWPsXOI3e+eBIDBHmaaD7uOkIWT/GnSBTvYr4NkF2xVxnmCsQ5eks13QdY6&lt;br /&gt;
MX1/gjwLl3nOPl6HBLc+KiHPLlMB rsa-key-20260325&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
You can use your mouse to highlight the entire string (starting at ssh-rsa and ending with the date) and copy it into your copy and paste buffer (this is actually a single line altogether). But first we want to do the following:&lt;br /&gt;
&lt;br /&gt;
* create a terminal connection to one of the login nodes (putty, or even ThinLinc)&lt;br /&gt;
* navigate to the folder &amp;quot;~/.ssh&amp;quot;&lt;br /&gt;
* edit or create a file called &amp;quot;authorized_keys&amp;quot;&lt;br /&gt;
* paste the key from puttygen into this file (it should be a single line of text)&lt;br /&gt;
* save the file&lt;br /&gt;
&lt;br /&gt;
==Using the private key to log in==&lt;br /&gt;
&lt;br /&gt;
The procedure is different for the various pieces of software that can be used to connect to the cluster.&lt;br /&gt;
&lt;br /&gt;
==Configuring PuTTY Connections==&lt;br /&gt;
&lt;br /&gt;
When creating or editing a PuTTY connection, connect your private key as shown here:&lt;br /&gt;
&lt;br /&gt;
[[File:Putty.png]]&lt;br /&gt;
&lt;br /&gt;
Once you save your connection, the key is stored for all future connections to this machine.&lt;br /&gt;
&lt;br /&gt;
==Configuring WinSCP==&lt;br /&gt;
&lt;br /&gt;
When creating or editing a WinSCP connection, connect your private key as shown here:&lt;br /&gt;
&lt;br /&gt;
[[File:Winscp.png]]&lt;br /&gt;
&lt;br /&gt;
Once you save your connection, the key is stored for all future connections to this machine.&lt;br /&gt;
&lt;br /&gt;
==Configuring FileZilla==&lt;br /&gt;
&lt;br /&gt;
When creating or editing a FileZilla connection (see Edit/Settings), connect your private key as shown here:&lt;br /&gt;
&lt;br /&gt;
[[File:Filezilla.png]]&lt;br /&gt;
&lt;br /&gt;
Once you save your connection, the key is stored for all future connections to this machine.&lt;/div&gt;</summary>
		<author><name>Ley</name></author>
	</entry>
	<entry>
		<id>https://wiki.anl.gov/wiki_tracc/index.php?title=File:Filezilla.png&amp;diff=2522</id>
		<title>File:Filezilla.png</title>
		<link rel="alternate" type="text/html" href="https://wiki.anl.gov/wiki_tracc/index.php?title=File:Filezilla.png&amp;diff=2522"/>
		<updated>2026-03-25T21:08:27Z</updated>

		<summary type="html">&lt;p&gt;Ley: &lt;/p&gt;
&lt;hr /&gt;
&lt;div&gt;&lt;/div&gt;</summary>
		<author><name>Ley</name></author>
	</entry>
	<entry>
		<id>https://wiki.anl.gov/wiki_tracc/index.php?title=Private/Public_Key_Authentication&amp;diff=2521</id>
		<title>Private/Public Key Authentication</title>
		<link rel="alternate" type="text/html" href="https://wiki.anl.gov/wiki_tracc/index.php?title=Private/Public_Key_Authentication&amp;diff=2521"/>
		<updated>2026-03-25T21:06:58Z</updated>

		<summary type="html">&lt;p&gt;Ley: /* Configuring WinSCP */&lt;/p&gt;
&lt;hr /&gt;
&lt;div&gt;==Creating a Private/Public Key Pair==&lt;br /&gt;
&lt;br /&gt;
If you have never use public key authentication before, you will have to start by creating a key pair. Once the key pair is created, you need to keep the private key in a safe location. Never upload the private key, and never send it by Email. Whoever has your private key can use it to try logging in to ARROW. So, treat the private key as a secret at all times.&lt;br /&gt;
&lt;br /&gt;
The easiest way to prepare a key pair is the use of &amp;quot;puttygen&amp;quot;, part of the &amp;quot;putty&amp;quot; software on Windows. Pressing the Windows Key together with R will open a window where you enter &amp;quot;puttygen&amp;quot;. Once you press &amp;quot;Ok&amp;quot;, the software opens up:&lt;br /&gt;
&lt;br /&gt;
[[File:Puttygen1.png]]&lt;br /&gt;
&lt;br /&gt;
We can go with the defaults and just click on the &amp;quot;Generate&amp;quot; button. Some randomness depends on you moving the mouse around on the screen. Once enough random data is gathered, the private and public key will be generated.&lt;br /&gt;
&lt;br /&gt;
[[File:Puttygen2.png]]&lt;br /&gt;
&lt;br /&gt;
Once the keys have been generated, you need to store the private key in a easy to find location on your local file system. I recommend creating a folder &amp;quot;Keys&amp;quot; within your &amp;quot;Documents&amp;quot; folder, but any other location is fine as well. The idea is that you may easily forget where the key is stored when you need it a later point in time. &lt;br /&gt;
&lt;br /&gt;
[[File:Puttygen3.png]]&lt;br /&gt;
&lt;br /&gt;
Click on &amp;quot;Save private key&amp;quot;, which will create the following dialog box for you:&lt;br /&gt;
&lt;br /&gt;
[[File:Puttygen4.png]]&lt;br /&gt;
&lt;br /&gt;
In an ideal world, assigning a password for your private key would be best, but that also means that you may have to enter that password every time you use the key, which can be a problem. Not assigning a password is one more reason to never risk losing this private key file.&lt;br /&gt;
&lt;br /&gt;
Navigate to the &amp;quot;Keys&amp;quot; folder in the &amp;quot;Documents: folder, and give the key a meaningful name. Something that tells you what it is used for, like &amp;quot;arrow-private&amp;quot;. The extension ppk is automatically added when the key is saved.&lt;br /&gt;
&lt;br /&gt;
Regarding the public key: It is not necessary to save the public key. When loading a private key, the public key is automatically displayed again at the top of the &amp;quot;puttygen&amp;quot; window. It looks somewhat like this:&lt;br /&gt;
&lt;br /&gt;
==Uploading the public key to one of the ARROW login nodes==&lt;br /&gt;
&lt;br /&gt;
The &amp;quot;puttygen&amp;quot; window still shows the public key at the top.&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
ssh-rsa AAAAB3NzaC1yc2EAfAADAQABAAABAQDOHgbwUYAAo+BMpvskvLw6eTqqs8OVUMSbAIobbC2/518PpJ/b&lt;br /&gt;
jW0n9NM4jlNJlEpuT79ZSv5RwSW8rP82qtcLdSEisxhXkyRLLjbzRWQSGxlVUI+K6MfZeFRYD+nqEH4Q1nSvxWJM&lt;br /&gt;
FigswLuWsygcJnGxv0FBGBP7pUrK5pz6AEv5Blf0MCuy0bvuIGhOYEMO4JeId6WYrAwJmZ3mjHqVwO3uQtrYP1Gi&lt;br /&gt;
71A+yk4x2hZepwW5fHiTbeCU/kDycqWPsXOI3e+eBIDBHmaaD7uOkIWT/GnSBTvYr4NkF2xVxnmCsQ5eks13QdY6&lt;br /&gt;
MX1/gjwLl3nOPl6HBLc+KiHPLlMB rsa-key-20260325&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
You can use your mouse to highlight the entire string (starting at ssh-rsa and ending with the date) and copy it into your copy and paste buffer (this is actually a single line altogether). But first we want to do the following:&lt;br /&gt;
&lt;br /&gt;
* create a terminal connection to one of the login nodes (putty, or even ThinLinc)&lt;br /&gt;
* navigate to the folder &amp;quot;~/.ssh&amp;quot;&lt;br /&gt;
* edit or create a file called &amp;quot;authorized_keys&amp;quot;&lt;br /&gt;
* paste the key from puttygen into this file (it should be a single line of text)&lt;br /&gt;
* save the file&lt;br /&gt;
&lt;br /&gt;
==Using the private key to log in==&lt;br /&gt;
&lt;br /&gt;
The procedure is different for the various pieces of software that can be used to connect to the cluster.&lt;br /&gt;
&lt;br /&gt;
==Configuring PuTTY Connections==&lt;br /&gt;
&lt;br /&gt;
When creating or editing a PuTTY connection, connect your private key as shown here:&lt;br /&gt;
&lt;br /&gt;
[[File:Putty.png]]&lt;br /&gt;
&lt;br /&gt;
Once you save your connection, the key is stored for all future connections to this machine.&lt;br /&gt;
&lt;br /&gt;
==Configuring WinSCP==&lt;br /&gt;
&lt;br /&gt;
When creating or editing a WinSCP connection, connect your private key as shown here:&lt;br /&gt;
&lt;br /&gt;
[[File:Winscp.png]]&lt;br /&gt;
&lt;br /&gt;
Once you save your connection, the key is stored for all future connections to this machine.&lt;br /&gt;
&lt;br /&gt;
==Configuring FileZilla==&lt;br /&gt;
&lt;br /&gt;
When creating or editing a FileZilla connection, connect your private key as shown here:&lt;br /&gt;
&lt;br /&gt;
[[File:Filezilla.png]]&lt;br /&gt;
&lt;br /&gt;
Once you save your connection, the key is stored for all future connections to this machine.&lt;/div&gt;</summary>
		<author><name>Ley</name></author>
	</entry>
	<entry>
		<id>https://wiki.anl.gov/wiki_tracc/index.php?title=Private/Public_Key_Authentication&amp;diff=2520</id>
		<title>Private/Public Key Authentication</title>
		<link rel="alternate" type="text/html" href="https://wiki.anl.gov/wiki_tracc/index.php?title=Private/Public_Key_Authentication&amp;diff=2520"/>
		<updated>2026-03-25T21:06:07Z</updated>

		<summary type="html">&lt;p&gt;Ley: /* Configuring PuTTY Connections */&lt;/p&gt;
&lt;hr /&gt;
&lt;div&gt;==Creating a Private/Public Key Pair==&lt;br /&gt;
&lt;br /&gt;
If you have never use public key authentication before, you will have to start by creating a key pair. Once the key pair is created, you need to keep the private key in a safe location. Never upload the private key, and never send it by Email. Whoever has your private key can use it to try logging in to ARROW. So, treat the private key as a secret at all times.&lt;br /&gt;
&lt;br /&gt;
The easiest way to prepare a key pair is the use of &amp;quot;puttygen&amp;quot;, part of the &amp;quot;putty&amp;quot; software on Windows. Pressing the Windows Key together with R will open a window where you enter &amp;quot;puttygen&amp;quot;. Once you press &amp;quot;Ok&amp;quot;, the software opens up:&lt;br /&gt;
&lt;br /&gt;
[[File:Puttygen1.png]]&lt;br /&gt;
&lt;br /&gt;
We can go with the defaults and just click on the &amp;quot;Generate&amp;quot; button. Some randomness depends on you moving the mouse around on the screen. Once enough random data is gathered, the private and public key will be generated.&lt;br /&gt;
&lt;br /&gt;
[[File:Puttygen2.png]]&lt;br /&gt;
&lt;br /&gt;
Once the keys have been generated, you need to store the private key in a easy to find location on your local file system. I recommend creating a folder &amp;quot;Keys&amp;quot; within your &amp;quot;Documents&amp;quot; folder, but any other location is fine as well. The idea is that you may easily forget where the key is stored when you need it a later point in time. &lt;br /&gt;
&lt;br /&gt;
[[File:Puttygen3.png]]&lt;br /&gt;
&lt;br /&gt;
Click on &amp;quot;Save private key&amp;quot;, which will create the following dialog box for you:&lt;br /&gt;
&lt;br /&gt;
[[File:Puttygen4.png]]&lt;br /&gt;
&lt;br /&gt;
In an ideal world, assigning a password for your private key would be best, but that also means that you may have to enter that password every time you use the key, which can be a problem. Not assigning a password is one more reason to never risk losing this private key file.&lt;br /&gt;
&lt;br /&gt;
Navigate to the &amp;quot;Keys&amp;quot; folder in the &amp;quot;Documents: folder, and give the key a meaningful name. Something that tells you what it is used for, like &amp;quot;arrow-private&amp;quot;. The extension ppk is automatically added when the key is saved.&lt;br /&gt;
&lt;br /&gt;
Regarding the public key: It is not necessary to save the public key. When loading a private key, the public key is automatically displayed again at the top of the &amp;quot;puttygen&amp;quot; window. It looks somewhat like this:&lt;br /&gt;
&lt;br /&gt;
==Uploading the public key to one of the ARROW login nodes==&lt;br /&gt;
&lt;br /&gt;
The &amp;quot;puttygen&amp;quot; window still shows the public key at the top.&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
ssh-rsa AAAAB3NzaC1yc2EAfAADAQABAAABAQDOHgbwUYAAo+BMpvskvLw6eTqqs8OVUMSbAIobbC2/518PpJ/b&lt;br /&gt;
jW0n9NM4jlNJlEpuT79ZSv5RwSW8rP82qtcLdSEisxhXkyRLLjbzRWQSGxlVUI+K6MfZeFRYD+nqEH4Q1nSvxWJM&lt;br /&gt;
FigswLuWsygcJnGxv0FBGBP7pUrK5pz6AEv5Blf0MCuy0bvuIGhOYEMO4JeId6WYrAwJmZ3mjHqVwO3uQtrYP1Gi&lt;br /&gt;
71A+yk4x2hZepwW5fHiTbeCU/kDycqWPsXOI3e+eBIDBHmaaD7uOkIWT/GnSBTvYr4NkF2xVxnmCsQ5eks13QdY6&lt;br /&gt;
MX1/gjwLl3nOPl6HBLc+KiHPLlMB rsa-key-20260325&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
You can use your mouse to highlight the entire string (starting at ssh-rsa and ending with the date) and copy it into your copy and paste buffer (this is actually a single line altogether). But first we want to do the following:&lt;br /&gt;
&lt;br /&gt;
* create a terminal connection to one of the login nodes (putty, or even ThinLinc)&lt;br /&gt;
* navigate to the folder &amp;quot;~/.ssh&amp;quot;&lt;br /&gt;
* edit or create a file called &amp;quot;authorized_keys&amp;quot;&lt;br /&gt;
* paste the key from puttygen into this file (it should be a single line of text)&lt;br /&gt;
* save the file&lt;br /&gt;
&lt;br /&gt;
==Using the private key to log in==&lt;br /&gt;
&lt;br /&gt;
The procedure is different for the various pieces of software that can be used to connect to the cluster.&lt;br /&gt;
&lt;br /&gt;
==Configuring PuTTY Connections==&lt;br /&gt;
&lt;br /&gt;
When creating or editing a PuTTY connection, connect your private key as shown here:&lt;br /&gt;
&lt;br /&gt;
[[File:Putty.png]]&lt;br /&gt;
&lt;br /&gt;
Once you save your connection, the key is stored for all future connections to this machine.&lt;br /&gt;
&lt;br /&gt;
==Configuring WinSCP==&lt;br /&gt;
&lt;br /&gt;
When creating or editing a WinSCP connection, connect your private key as shown here:&lt;br /&gt;
&lt;br /&gt;
[[File:Winscp.png]]&lt;br /&gt;
&lt;br /&gt;
Once you save your connection, the key is stored for all future connections to this machine.&lt;/div&gt;</summary>
		<author><name>Ley</name></author>
	</entry>
	<entry>
		<id>https://wiki.anl.gov/wiki_tracc/index.php?title=Private/Public_Key_Authentication&amp;diff=2519</id>
		<title>Private/Public Key Authentication</title>
		<link rel="alternate" type="text/html" href="https://wiki.anl.gov/wiki_tracc/index.php?title=Private/Public_Key_Authentication&amp;diff=2519"/>
		<updated>2026-03-25T21:05:54Z</updated>

		<summary type="html">&lt;p&gt;Ley: /* Configuring WinSCP */&lt;/p&gt;
&lt;hr /&gt;
&lt;div&gt;==Creating a Private/Public Key Pair==&lt;br /&gt;
&lt;br /&gt;
If you have never use public key authentication before, you will have to start by creating a key pair. Once the key pair is created, you need to keep the private key in a safe location. Never upload the private key, and never send it by Email. Whoever has your private key can use it to try logging in to ARROW. So, treat the private key as a secret at all times.&lt;br /&gt;
&lt;br /&gt;
The easiest way to prepare a key pair is the use of &amp;quot;puttygen&amp;quot;, part of the &amp;quot;putty&amp;quot; software on Windows. Pressing the Windows Key together with R will open a window where you enter &amp;quot;puttygen&amp;quot;. Once you press &amp;quot;Ok&amp;quot;, the software opens up:&lt;br /&gt;
&lt;br /&gt;
[[File:Puttygen1.png]]&lt;br /&gt;
&lt;br /&gt;
We can go with the defaults and just click on the &amp;quot;Generate&amp;quot; button. Some randomness depends on you moving the mouse around on the screen. Once enough random data is gathered, the private and public key will be generated.&lt;br /&gt;
&lt;br /&gt;
[[File:Puttygen2.png]]&lt;br /&gt;
&lt;br /&gt;
Once the keys have been generated, you need to store the private key in a easy to find location on your local file system. I recommend creating a folder &amp;quot;Keys&amp;quot; within your &amp;quot;Documents&amp;quot; folder, but any other location is fine as well. The idea is that you may easily forget where the key is stored when you need it a later point in time. &lt;br /&gt;
&lt;br /&gt;
[[File:Puttygen3.png]]&lt;br /&gt;
&lt;br /&gt;
Click on &amp;quot;Save private key&amp;quot;, which will create the following dialog box for you:&lt;br /&gt;
&lt;br /&gt;
[[File:Puttygen4.png]]&lt;br /&gt;
&lt;br /&gt;
In an ideal world, assigning a password for your private key would be best, but that also means that you may have to enter that password every time you use the key, which can be a problem. Not assigning a password is one more reason to never risk losing this private key file.&lt;br /&gt;
&lt;br /&gt;
Navigate to the &amp;quot;Keys&amp;quot; folder in the &amp;quot;Documents: folder, and give the key a meaningful name. Something that tells you what it is used for, like &amp;quot;arrow-private&amp;quot;. The extension ppk is automatically added when the key is saved.&lt;br /&gt;
&lt;br /&gt;
Regarding the public key: It is not necessary to save the public key. When loading a private key, the public key is automatically displayed again at the top of the &amp;quot;puttygen&amp;quot; window. It looks somewhat like this:&lt;br /&gt;
&lt;br /&gt;
==Uploading the public key to one of the ARROW login nodes==&lt;br /&gt;
&lt;br /&gt;
The &amp;quot;puttygen&amp;quot; window still shows the public key at the top.&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
ssh-rsa AAAAB3NzaC1yc2EAfAADAQABAAABAQDOHgbwUYAAo+BMpvskvLw6eTqqs8OVUMSbAIobbC2/518PpJ/b&lt;br /&gt;
jW0n9NM4jlNJlEpuT79ZSv5RwSW8rP82qtcLdSEisxhXkyRLLjbzRWQSGxlVUI+K6MfZeFRYD+nqEH4Q1nSvxWJM&lt;br /&gt;
FigswLuWsygcJnGxv0FBGBP7pUrK5pz6AEv5Blf0MCuy0bvuIGhOYEMO4JeId6WYrAwJmZ3mjHqVwO3uQtrYP1Gi&lt;br /&gt;
71A+yk4x2hZepwW5fHiTbeCU/kDycqWPsXOI3e+eBIDBHmaaD7uOkIWT/GnSBTvYr4NkF2xVxnmCsQ5eks13QdY6&lt;br /&gt;
MX1/gjwLl3nOPl6HBLc+KiHPLlMB rsa-key-20260325&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
You can use your mouse to highlight the entire string (starting at ssh-rsa and ending with the date) and copy it into your copy and paste buffer (this is actually a single line altogether). But first we want to do the following:&lt;br /&gt;
&lt;br /&gt;
* create a terminal connection to one of the login nodes (putty, or even ThinLinc)&lt;br /&gt;
* navigate to the folder &amp;quot;~/.ssh&amp;quot;&lt;br /&gt;
* edit or create a file called &amp;quot;authorized_keys&amp;quot;&lt;br /&gt;
* paste the key from puttygen into this file (it should be a single line of text)&lt;br /&gt;
* save the file&lt;br /&gt;
&lt;br /&gt;
==Using the private key to log in==&lt;br /&gt;
&lt;br /&gt;
The procedure is different for the various pieces of software that can be used to connect to the cluster.&lt;br /&gt;
&lt;br /&gt;
==Configuring PuTTY Connections==&lt;br /&gt;
&lt;br /&gt;
When creating or editing a PuTTY, connect your private key as shown here:&lt;br /&gt;
&lt;br /&gt;
[[File:Putty.png]]&lt;br /&gt;
&lt;br /&gt;
Once you save your connection, the key is stored for all future connections to this machine.&lt;br /&gt;
&lt;br /&gt;
==Configuring WinSCP==&lt;br /&gt;
&lt;br /&gt;
When creating or editing a WinSCP connection, connect your private key as shown here:&lt;br /&gt;
&lt;br /&gt;
[[File:Winscp.png]]&lt;br /&gt;
&lt;br /&gt;
Once you save your connection, the key is stored for all future connections to this machine.&lt;/div&gt;</summary>
		<author><name>Ley</name></author>
	</entry>
	<entry>
		<id>https://wiki.anl.gov/wiki_tracc/index.php?title=File:Winscp.png&amp;diff=2518</id>
		<title>File:Winscp.png</title>
		<link rel="alternate" type="text/html" href="https://wiki.anl.gov/wiki_tracc/index.php?title=File:Winscp.png&amp;diff=2518"/>
		<updated>2026-03-25T21:04:27Z</updated>

		<summary type="html">&lt;p&gt;Ley: &lt;/p&gt;
&lt;hr /&gt;
&lt;div&gt;&lt;/div&gt;</summary>
		<author><name>Ley</name></author>
	</entry>
	<entry>
		<id>https://wiki.anl.gov/wiki_tracc/index.php?title=Private/Public_Key_Authentication&amp;diff=2517</id>
		<title>Private/Public Key Authentication</title>
		<link rel="alternate" type="text/html" href="https://wiki.anl.gov/wiki_tracc/index.php?title=Private/Public_Key_Authentication&amp;diff=2517"/>
		<updated>2026-03-25T21:03:54Z</updated>

		<summary type="html">&lt;p&gt;Ley: /* Configuring WinSCP */&lt;/p&gt;
&lt;hr /&gt;
&lt;div&gt;==Creating a Private/Public Key Pair==&lt;br /&gt;
&lt;br /&gt;
If you have never use public key authentication before, you will have to start by creating a key pair. Once the key pair is created, you need to keep the private key in a safe location. Never upload the private key, and never send it by Email. Whoever has your private key can use it to try logging in to ARROW. So, treat the private key as a secret at all times.&lt;br /&gt;
&lt;br /&gt;
The easiest way to prepare a key pair is the use of &amp;quot;puttygen&amp;quot;, part of the &amp;quot;putty&amp;quot; software on Windows. Pressing the Windows Key together with R will open a window where you enter &amp;quot;puttygen&amp;quot;. Once you press &amp;quot;Ok&amp;quot;, the software opens up:&lt;br /&gt;
&lt;br /&gt;
[[File:Puttygen1.png]]&lt;br /&gt;
&lt;br /&gt;
We can go with the defaults and just click on the &amp;quot;Generate&amp;quot; button. Some randomness depends on you moving the mouse around on the screen. Once enough random data is gathered, the private and public key will be generated.&lt;br /&gt;
&lt;br /&gt;
[[File:Puttygen2.png]]&lt;br /&gt;
&lt;br /&gt;
Once the keys have been generated, you need to store the private key in a easy to find location on your local file system. I recommend creating a folder &amp;quot;Keys&amp;quot; within your &amp;quot;Documents&amp;quot; folder, but any other location is fine as well. The idea is that you may easily forget where the key is stored when you need it a later point in time. &lt;br /&gt;
&lt;br /&gt;
[[File:Puttygen3.png]]&lt;br /&gt;
&lt;br /&gt;
Click on &amp;quot;Save private key&amp;quot;, which will create the following dialog box for you:&lt;br /&gt;
&lt;br /&gt;
[[File:Puttygen4.png]]&lt;br /&gt;
&lt;br /&gt;
In an ideal world, assigning a password for your private key would be best, but that also means that you may have to enter that password every time you use the key, which can be a problem. Not assigning a password is one more reason to never risk losing this private key file.&lt;br /&gt;
&lt;br /&gt;
Navigate to the &amp;quot;Keys&amp;quot; folder in the &amp;quot;Documents: folder, and give the key a meaningful name. Something that tells you what it is used for, like &amp;quot;arrow-private&amp;quot;. The extension ppk is automatically added when the key is saved.&lt;br /&gt;
&lt;br /&gt;
Regarding the public key: It is not necessary to save the public key. When loading a private key, the public key is automatically displayed again at the top of the &amp;quot;puttygen&amp;quot; window. It looks somewhat like this:&lt;br /&gt;
&lt;br /&gt;
==Uploading the public key to one of the ARROW login nodes==&lt;br /&gt;
&lt;br /&gt;
The &amp;quot;puttygen&amp;quot; window still shows the public key at the top.&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
ssh-rsa AAAAB3NzaC1yc2EAfAADAQABAAABAQDOHgbwUYAAo+BMpvskvLw6eTqqs8OVUMSbAIobbC2/518PpJ/b&lt;br /&gt;
jW0n9NM4jlNJlEpuT79ZSv5RwSW8rP82qtcLdSEisxhXkyRLLjbzRWQSGxlVUI+K6MfZeFRYD+nqEH4Q1nSvxWJM&lt;br /&gt;
FigswLuWsygcJnGxv0FBGBP7pUrK5pz6AEv5Blf0MCuy0bvuIGhOYEMO4JeId6WYrAwJmZ3mjHqVwO3uQtrYP1Gi&lt;br /&gt;
71A+yk4x2hZepwW5fHiTbeCU/kDycqWPsXOI3e+eBIDBHmaaD7uOkIWT/GnSBTvYr4NkF2xVxnmCsQ5eks13QdY6&lt;br /&gt;
MX1/gjwLl3nOPl6HBLc+KiHPLlMB rsa-key-20260325&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
You can use your mouse to highlight the entire string (starting at ssh-rsa and ending with the date) and copy it into your copy and paste buffer (this is actually a single line altogether). But first we want to do the following:&lt;br /&gt;
&lt;br /&gt;
* create a terminal connection to one of the login nodes (putty, or even ThinLinc)&lt;br /&gt;
* navigate to the folder &amp;quot;~/.ssh&amp;quot;&lt;br /&gt;
* edit or create a file called &amp;quot;authorized_keys&amp;quot;&lt;br /&gt;
* paste the key from puttygen into this file (it should be a single line of text)&lt;br /&gt;
* save the file&lt;br /&gt;
&lt;br /&gt;
==Using the private key to log in==&lt;br /&gt;
&lt;br /&gt;
The procedure is different for the various pieces of software that can be used to connect to the cluster.&lt;br /&gt;
&lt;br /&gt;
==Configuring PuTTY Connections==&lt;br /&gt;
&lt;br /&gt;
When creating or editing a PuTTY, connect your private key as shown here:&lt;br /&gt;
&lt;br /&gt;
[[File:Putty.png]]&lt;br /&gt;
&lt;br /&gt;
Once you save your connection, the key is stored for all future connections to this machine.&lt;br /&gt;
&lt;br /&gt;
==Configuring WinSCP==&lt;br /&gt;
&lt;br /&gt;
When creating or editing a PuTTY, connect your private key as shown here:&lt;br /&gt;
&lt;br /&gt;
[[File:Winscp.png]]&lt;br /&gt;
&lt;br /&gt;
Once you save your connection, the key is stored for all future connections to this machine.&lt;/div&gt;</summary>
		<author><name>Ley</name></author>
	</entry>
	<entry>
		<id>https://wiki.anl.gov/wiki_tracc/index.php?title=Private/Public_Key_Authentication&amp;diff=2516</id>
		<title>Private/Public Key Authentication</title>
		<link rel="alternate" type="text/html" href="https://wiki.anl.gov/wiki_tracc/index.php?title=Private/Public_Key_Authentication&amp;diff=2516"/>
		<updated>2026-03-25T21:03:26Z</updated>

		<summary type="html">&lt;p&gt;Ley: /* Configuring PuTTY Connections */&lt;/p&gt;
&lt;hr /&gt;
&lt;div&gt;==Creating a Private/Public Key Pair==&lt;br /&gt;
&lt;br /&gt;
If you have never use public key authentication before, you will have to start by creating a key pair. Once the key pair is created, you need to keep the private key in a safe location. Never upload the private key, and never send it by Email. Whoever has your private key can use it to try logging in to ARROW. So, treat the private key as a secret at all times.&lt;br /&gt;
&lt;br /&gt;
The easiest way to prepare a key pair is the use of &amp;quot;puttygen&amp;quot;, part of the &amp;quot;putty&amp;quot; software on Windows. Pressing the Windows Key together with R will open a window where you enter &amp;quot;puttygen&amp;quot;. Once you press &amp;quot;Ok&amp;quot;, the software opens up:&lt;br /&gt;
&lt;br /&gt;
[[File:Puttygen1.png]]&lt;br /&gt;
&lt;br /&gt;
We can go with the defaults and just click on the &amp;quot;Generate&amp;quot; button. Some randomness depends on you moving the mouse around on the screen. Once enough random data is gathered, the private and public key will be generated.&lt;br /&gt;
&lt;br /&gt;
[[File:Puttygen2.png]]&lt;br /&gt;
&lt;br /&gt;
Once the keys have been generated, you need to store the private key in a easy to find location on your local file system. I recommend creating a folder &amp;quot;Keys&amp;quot; within your &amp;quot;Documents&amp;quot; folder, but any other location is fine as well. The idea is that you may easily forget where the key is stored when you need it a later point in time. &lt;br /&gt;
&lt;br /&gt;
[[File:Puttygen3.png]]&lt;br /&gt;
&lt;br /&gt;
Click on &amp;quot;Save private key&amp;quot;, which will create the following dialog box for you:&lt;br /&gt;
&lt;br /&gt;
[[File:Puttygen4.png]]&lt;br /&gt;
&lt;br /&gt;
In an ideal world, assigning a password for your private key would be best, but that also means that you may have to enter that password every time you use the key, which can be a problem. Not assigning a password is one more reason to never risk losing this private key file.&lt;br /&gt;
&lt;br /&gt;
Navigate to the &amp;quot;Keys&amp;quot; folder in the &amp;quot;Documents: folder, and give the key a meaningful name. Something that tells you what it is used for, like &amp;quot;arrow-private&amp;quot;. The extension ppk is automatically added when the key is saved.&lt;br /&gt;
&lt;br /&gt;
Regarding the public key: It is not necessary to save the public key. When loading a private key, the public key is automatically displayed again at the top of the &amp;quot;puttygen&amp;quot; window. It looks somewhat like this:&lt;br /&gt;
&lt;br /&gt;
==Uploading the public key to one of the ARROW login nodes==&lt;br /&gt;
&lt;br /&gt;
The &amp;quot;puttygen&amp;quot; window still shows the public key at the top.&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
ssh-rsa AAAAB3NzaC1yc2EAfAADAQABAAABAQDOHgbwUYAAo+BMpvskvLw6eTqqs8OVUMSbAIobbC2/518PpJ/b&lt;br /&gt;
jW0n9NM4jlNJlEpuT79ZSv5RwSW8rP82qtcLdSEisxhXkyRLLjbzRWQSGxlVUI+K6MfZeFRYD+nqEH4Q1nSvxWJM&lt;br /&gt;
FigswLuWsygcJnGxv0FBGBP7pUrK5pz6AEv5Blf0MCuy0bvuIGhOYEMO4JeId6WYrAwJmZ3mjHqVwO3uQtrYP1Gi&lt;br /&gt;
71A+yk4x2hZepwW5fHiTbeCU/kDycqWPsXOI3e+eBIDBHmaaD7uOkIWT/GnSBTvYr4NkF2xVxnmCsQ5eks13QdY6&lt;br /&gt;
MX1/gjwLl3nOPl6HBLc+KiHPLlMB rsa-key-20260325&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
You can use your mouse to highlight the entire string (starting at ssh-rsa and ending with the date) and copy it into your copy and paste buffer (this is actually a single line altogether). But first we want to do the following:&lt;br /&gt;
&lt;br /&gt;
* create a terminal connection to one of the login nodes (putty, or even ThinLinc)&lt;br /&gt;
* navigate to the folder &amp;quot;~/.ssh&amp;quot;&lt;br /&gt;
* edit or create a file called &amp;quot;authorized_keys&amp;quot;&lt;br /&gt;
* paste the key from puttygen into this file (it should be a single line of text)&lt;br /&gt;
* save the file&lt;br /&gt;
&lt;br /&gt;
==Using the private key to log in==&lt;br /&gt;
&lt;br /&gt;
The procedure is different for the various pieces of software that can be used to connect to the cluster.&lt;br /&gt;
&lt;br /&gt;
==Configuring PuTTY Connections==&lt;br /&gt;
&lt;br /&gt;
When creating or editing a PuTTY, connect your private key as shown here:&lt;br /&gt;
&lt;br /&gt;
[[File:Putty.png]]&lt;br /&gt;
&lt;br /&gt;
Once you save your connection, the key is stored for all future connections to this machine.&lt;br /&gt;
&lt;br /&gt;
==Configuring WinSCP==&lt;/div&gt;</summary>
		<author><name>Ley</name></author>
	</entry>
	<entry>
		<id>https://wiki.anl.gov/wiki_tracc/index.php?title=Private/Public_Key_Authentication&amp;diff=2515</id>
		<title>Private/Public Key Authentication</title>
		<link rel="alternate" type="text/html" href="https://wiki.anl.gov/wiki_tracc/index.php?title=Private/Public_Key_Authentication&amp;diff=2515"/>
		<updated>2026-03-25T21:01:32Z</updated>

		<summary type="html">&lt;p&gt;Ley: /* putty */&lt;/p&gt;
&lt;hr /&gt;
&lt;div&gt;==Creating a Private/Public Key Pair==&lt;br /&gt;
&lt;br /&gt;
If you have never use public key authentication before, you will have to start by creating a key pair. Once the key pair is created, you need to keep the private key in a safe location. Never upload the private key, and never send it by Email. Whoever has your private key can use it to try logging in to ARROW. So, treat the private key as a secret at all times.&lt;br /&gt;
&lt;br /&gt;
The easiest way to prepare a key pair is the use of &amp;quot;puttygen&amp;quot;, part of the &amp;quot;putty&amp;quot; software on Windows. Pressing the Windows Key together with R will open a window where you enter &amp;quot;puttygen&amp;quot;. Once you press &amp;quot;Ok&amp;quot;, the software opens up:&lt;br /&gt;
&lt;br /&gt;
[[File:Puttygen1.png]]&lt;br /&gt;
&lt;br /&gt;
We can go with the defaults and just click on the &amp;quot;Generate&amp;quot; button. Some randomness depends on you moving the mouse around on the screen. Once enough random data is gathered, the private and public key will be generated.&lt;br /&gt;
&lt;br /&gt;
[[File:Puttygen2.png]]&lt;br /&gt;
&lt;br /&gt;
Once the keys have been generated, you need to store the private key in a easy to find location on your local file system. I recommend creating a folder &amp;quot;Keys&amp;quot; within your &amp;quot;Documents&amp;quot; folder, but any other location is fine as well. The idea is that you may easily forget where the key is stored when you need it a later point in time. &lt;br /&gt;
&lt;br /&gt;
[[File:Puttygen3.png]]&lt;br /&gt;
&lt;br /&gt;
Click on &amp;quot;Save private key&amp;quot;, which will create the following dialog box for you:&lt;br /&gt;
&lt;br /&gt;
[[File:Puttygen4.png]]&lt;br /&gt;
&lt;br /&gt;
In an ideal world, assigning a password for your private key would be best, but that also means that you may have to enter that password every time you use the key, which can be a problem. Not assigning a password is one more reason to never risk losing this private key file.&lt;br /&gt;
&lt;br /&gt;
Navigate to the &amp;quot;Keys&amp;quot; folder in the &amp;quot;Documents: folder, and give the key a meaningful name. Something that tells you what it is used for, like &amp;quot;arrow-private&amp;quot;. The extension ppk is automatically added when the key is saved.&lt;br /&gt;
&lt;br /&gt;
Regarding the public key: It is not necessary to save the public key. When loading a private key, the public key is automatically displayed again at the top of the &amp;quot;puttygen&amp;quot; window. It looks somewhat like this:&lt;br /&gt;
&lt;br /&gt;
==Uploading the public key to one of the ARROW login nodes==&lt;br /&gt;
&lt;br /&gt;
The &amp;quot;puttygen&amp;quot; window still shows the public key at the top.&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
ssh-rsa AAAAB3NzaC1yc2EAfAADAQABAAABAQDOHgbwUYAAo+BMpvskvLw6eTqqs8OVUMSbAIobbC2/518PpJ/b&lt;br /&gt;
jW0n9NM4jlNJlEpuT79ZSv5RwSW8rP82qtcLdSEisxhXkyRLLjbzRWQSGxlVUI+K6MfZeFRYD+nqEH4Q1nSvxWJM&lt;br /&gt;
FigswLuWsygcJnGxv0FBGBP7pUrK5pz6AEv5Blf0MCuy0bvuIGhOYEMO4JeId6WYrAwJmZ3mjHqVwO3uQtrYP1Gi&lt;br /&gt;
71A+yk4x2hZepwW5fHiTbeCU/kDycqWPsXOI3e+eBIDBHmaaD7uOkIWT/GnSBTvYr4NkF2xVxnmCsQ5eks13QdY6&lt;br /&gt;
MX1/gjwLl3nOPl6HBLc+KiHPLlMB rsa-key-20260325&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
You can use your mouse to highlight the entire string (starting at ssh-rsa and ending with the date) and copy it into your copy and paste buffer (this is actually a single line altogether). But first we want to do the following:&lt;br /&gt;
&lt;br /&gt;
* create a terminal connection to one of the login nodes (putty, or even ThinLinc)&lt;br /&gt;
* navigate to the folder &amp;quot;~/.ssh&amp;quot;&lt;br /&gt;
* edit or create a file called &amp;quot;authorized_keys&amp;quot;&lt;br /&gt;
* paste the key from puttygen into this file (it should be a single line of text)&lt;br /&gt;
* save the file&lt;br /&gt;
&lt;br /&gt;
==Using the private key to log in==&lt;br /&gt;
&lt;br /&gt;
The procedure is different for the various pieces of software that can be used to connect to the cluster.&lt;br /&gt;
&lt;br /&gt;
==Configuring PuTTY Connections==&lt;br /&gt;
&lt;br /&gt;
When creating or editing a PuTTY, connect your private key as shown here:&lt;br /&gt;
&lt;br /&gt;
[[File:Putty.png]]&lt;br /&gt;
&lt;br /&gt;
Once you save your connection, the key is stored for all future connections to this machine.&lt;/div&gt;</summary>
		<author><name>Ley</name></author>
	</entry>
	<entry>
		<id>https://wiki.anl.gov/wiki_tracc/index.php?title=Private/Public_Key_Authentication&amp;diff=2514</id>
		<title>Private/Public Key Authentication</title>
		<link rel="alternate" type="text/html" href="https://wiki.anl.gov/wiki_tracc/index.php?title=Private/Public_Key_Authentication&amp;diff=2514"/>
		<updated>2026-03-25T20:06:10Z</updated>

		<summary type="html">&lt;p&gt;Ley: /* putty */&lt;/p&gt;
&lt;hr /&gt;
&lt;div&gt;==Creating a Private/Public Key Pair==&lt;br /&gt;
&lt;br /&gt;
If you have never use public key authentication before, you will have to start by creating a key pair. Once the key pair is created, you need to keep the private key in a safe location. Never upload the private key, and never send it by Email. Whoever has your private key can use it to try logging in to ARROW. So, treat the private key as a secret at all times.&lt;br /&gt;
&lt;br /&gt;
The easiest way to prepare a key pair is the use of &amp;quot;puttygen&amp;quot;, part of the &amp;quot;putty&amp;quot; software on Windows. Pressing the Windows Key together with R will open a window where you enter &amp;quot;puttygen&amp;quot;. Once you press &amp;quot;Ok&amp;quot;, the software opens up:&lt;br /&gt;
&lt;br /&gt;
[[File:Puttygen1.png]]&lt;br /&gt;
&lt;br /&gt;
We can go with the defaults and just click on the &amp;quot;Generate&amp;quot; button. Some randomness depends on you moving the mouse around on the screen. Once enough random data is gathered, the private and public key will be generated.&lt;br /&gt;
&lt;br /&gt;
[[File:Puttygen2.png]]&lt;br /&gt;
&lt;br /&gt;
Once the keys have been generated, you need to store the private key in a easy to find location on your local file system. I recommend creating a folder &amp;quot;Keys&amp;quot; within your &amp;quot;Documents&amp;quot; folder, but any other location is fine as well. The idea is that you may easily forget where the key is stored when you need it a later point in time. &lt;br /&gt;
&lt;br /&gt;
[[File:Puttygen3.png]]&lt;br /&gt;
&lt;br /&gt;
Click on &amp;quot;Save private key&amp;quot;, which will create the following dialog box for you:&lt;br /&gt;
&lt;br /&gt;
[[File:Puttygen4.png]]&lt;br /&gt;
&lt;br /&gt;
In an ideal world, assigning a password for your private key would be best, but that also means that you may have to enter that password every time you use the key, which can be a problem. Not assigning a password is one more reason to never risk losing this private key file.&lt;br /&gt;
&lt;br /&gt;
Navigate to the &amp;quot;Keys&amp;quot; folder in the &amp;quot;Documents: folder, and give the key a meaningful name. Something that tells you what it is used for, like &amp;quot;arrow-private&amp;quot;. The extension ppk is automatically added when the key is saved.&lt;br /&gt;
&lt;br /&gt;
Regarding the public key: It is not necessary to save the public key. When loading a private key, the public key is automatically displayed again at the top of the &amp;quot;puttygen&amp;quot; window. It looks somewhat like this:&lt;br /&gt;
&lt;br /&gt;
==Uploading the public key to one of the ARROW login nodes==&lt;br /&gt;
&lt;br /&gt;
The &amp;quot;puttygen&amp;quot; window still shows the public key at the top.&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
ssh-rsa AAAAB3NzaC1yc2EAfAADAQABAAABAQDOHgbwUYAAo+BMpvskvLw6eTqqs8OVUMSbAIobbC2/518PpJ/b&lt;br /&gt;
jW0n9NM4jlNJlEpuT79ZSv5RwSW8rP82qtcLdSEisxhXkyRLLjbzRWQSGxlVUI+K6MfZeFRYD+nqEH4Q1nSvxWJM&lt;br /&gt;
FigswLuWsygcJnGxv0FBGBP7pUrK5pz6AEv5Blf0MCuy0bvuIGhOYEMO4JeId6WYrAwJmZ3mjHqVwO3uQtrYP1Gi&lt;br /&gt;
71A+yk4x2hZepwW5fHiTbeCU/kDycqWPsXOI3e+eBIDBHmaaD7uOkIWT/GnSBTvYr4NkF2xVxnmCsQ5eks13QdY6&lt;br /&gt;
MX1/gjwLl3nOPl6HBLc+KiHPLlMB rsa-key-20260325&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
You can use your mouse to highlight the entire string (starting at ssh-rsa and ending with the date) and copy it into your copy and paste buffer (this is actually a single line altogether). But first we want to do the following:&lt;br /&gt;
&lt;br /&gt;
* create a terminal connection to one of the login nodes (putty, or even ThinLinc)&lt;br /&gt;
* navigate to the folder &amp;quot;~/.ssh&amp;quot;&lt;br /&gt;
* edit or create a file called &amp;quot;authorized_keys&amp;quot;&lt;br /&gt;
* paste the key from puttygen into this file (it should be a single line of text)&lt;br /&gt;
* save the file&lt;br /&gt;
&lt;br /&gt;
==Using the private key to log in==&lt;br /&gt;
&lt;br /&gt;
The procedure is different for the various pieces of software that can be used to connect to the cluster.&lt;br /&gt;
&lt;br /&gt;
===putty===&lt;br /&gt;
&lt;br /&gt;
[[File:Putty.png]]&lt;/div&gt;</summary>
		<author><name>Ley</name></author>
	</entry>
	<entry>
		<id>https://wiki.anl.gov/wiki_tracc/index.php?title=File:Putty.png&amp;diff=2513</id>
		<title>File:Putty.png</title>
		<link rel="alternate" type="text/html" href="https://wiki.anl.gov/wiki_tracc/index.php?title=File:Putty.png&amp;diff=2513"/>
		<updated>2026-03-25T20:04:28Z</updated>

		<summary type="html">&lt;p&gt;Ley: &lt;/p&gt;
&lt;hr /&gt;
&lt;div&gt;&lt;/div&gt;</summary>
		<author><name>Ley</name></author>
	</entry>
	<entry>
		<id>https://wiki.anl.gov/wiki_tracc/index.php?title=Private/Public_Key_Authentication&amp;diff=2512</id>
		<title>Private/Public Key Authentication</title>
		<link rel="alternate" type="text/html" href="https://wiki.anl.gov/wiki_tracc/index.php?title=Private/Public_Key_Authentication&amp;diff=2512"/>
		<updated>2026-03-25T20:03:13Z</updated>

		<summary type="html">&lt;p&gt;Ley: &lt;/p&gt;
&lt;hr /&gt;
&lt;div&gt;==Creating a Private/Public Key Pair==&lt;br /&gt;
&lt;br /&gt;
If you have never use public key authentication before, you will have to start by creating a key pair. Once the key pair is created, you need to keep the private key in a safe location. Never upload the private key, and never send it by Email. Whoever has your private key can use it to try logging in to ARROW. So, treat the private key as a secret at all times.&lt;br /&gt;
&lt;br /&gt;
The easiest way to prepare a key pair is the use of &amp;quot;puttygen&amp;quot;, part of the &amp;quot;putty&amp;quot; software on Windows. Pressing the Windows Key together with R will open a window where you enter &amp;quot;puttygen&amp;quot;. Once you press &amp;quot;Ok&amp;quot;, the software opens up:&lt;br /&gt;
&lt;br /&gt;
[[File:Puttygen1.png]]&lt;br /&gt;
&lt;br /&gt;
We can go with the defaults and just click on the &amp;quot;Generate&amp;quot; button. Some randomness depends on you moving the mouse around on the screen. Once enough random data is gathered, the private and public key will be generated.&lt;br /&gt;
&lt;br /&gt;
[[File:Puttygen2.png]]&lt;br /&gt;
&lt;br /&gt;
Once the keys have been generated, you need to store the private key in a easy to find location on your local file system. I recommend creating a folder &amp;quot;Keys&amp;quot; within your &amp;quot;Documents&amp;quot; folder, but any other location is fine as well. The idea is that you may easily forget where the key is stored when you need it a later point in time. &lt;br /&gt;
&lt;br /&gt;
[[File:Puttygen3.png]]&lt;br /&gt;
&lt;br /&gt;
Click on &amp;quot;Save private key&amp;quot;, which will create the following dialog box for you:&lt;br /&gt;
&lt;br /&gt;
[[File:Puttygen4.png]]&lt;br /&gt;
&lt;br /&gt;
In an ideal world, assigning a password for your private key would be best, but that also means that you may have to enter that password every time you use the key, which can be a problem. Not assigning a password is one more reason to never risk losing this private key file.&lt;br /&gt;
&lt;br /&gt;
Navigate to the &amp;quot;Keys&amp;quot; folder in the &amp;quot;Documents: folder, and give the key a meaningful name. Something that tells you what it is used for, like &amp;quot;arrow-private&amp;quot;. The extension ppk is automatically added when the key is saved.&lt;br /&gt;
&lt;br /&gt;
Regarding the public key: It is not necessary to save the public key. When loading a private key, the public key is automatically displayed again at the top of the &amp;quot;puttygen&amp;quot; window. It looks somewhat like this:&lt;br /&gt;
&lt;br /&gt;
==Uploading the public key to one of the ARROW login nodes==&lt;br /&gt;
&lt;br /&gt;
The &amp;quot;puttygen&amp;quot; window still shows the public key at the top.&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
ssh-rsa AAAAB3NzaC1yc2EAfAADAQABAAABAQDOHgbwUYAAo+BMpvskvLw6eTqqs8OVUMSbAIobbC2/518PpJ/b&lt;br /&gt;
jW0n9NM4jlNJlEpuT79ZSv5RwSW8rP82qtcLdSEisxhXkyRLLjbzRWQSGxlVUI+K6MfZeFRYD+nqEH4Q1nSvxWJM&lt;br /&gt;
FigswLuWsygcJnGxv0FBGBP7pUrK5pz6AEv5Blf0MCuy0bvuIGhOYEMO4JeId6WYrAwJmZ3mjHqVwO3uQtrYP1Gi&lt;br /&gt;
71A+yk4x2hZepwW5fHiTbeCU/kDycqWPsXOI3e+eBIDBHmaaD7uOkIWT/GnSBTvYr4NkF2xVxnmCsQ5eks13QdY6&lt;br /&gt;
MX1/gjwLl3nOPl6HBLc+KiHPLlMB rsa-key-20260325&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
You can use your mouse to highlight the entire string (starting at ssh-rsa and ending with the date) and copy it into your copy and paste buffer (this is actually a single line altogether). But first we want to do the following:&lt;br /&gt;
&lt;br /&gt;
* create a terminal connection to one of the login nodes (putty, or even ThinLinc)&lt;br /&gt;
* navigate to the folder &amp;quot;~/.ssh&amp;quot;&lt;br /&gt;
* edit or create a file called &amp;quot;authorized_keys&amp;quot;&lt;br /&gt;
* paste the key from puttygen into this file (it should be a single line of text)&lt;br /&gt;
* save the file&lt;br /&gt;
&lt;br /&gt;
==Using the private key to log in==&lt;br /&gt;
&lt;br /&gt;
The procedure is different for the various pieces of software that can be used to connect to the cluster.&lt;br /&gt;
&lt;br /&gt;
===putty===&lt;/div&gt;</summary>
		<author><name>Ley</name></author>
	</entry>
	<entry>
		<id>https://wiki.anl.gov/wiki_tracc/index.php?title=Private/Public_Key_Authentication&amp;diff=2511</id>
		<title>Private/Public Key Authentication</title>
		<link rel="alternate" type="text/html" href="https://wiki.anl.gov/wiki_tracc/index.php?title=Private/Public_Key_Authentication&amp;diff=2511"/>
		<updated>2026-03-25T19:56:51Z</updated>

		<summary type="html">&lt;p&gt;Ley: /* Creating a Private/Public Key Pair */&lt;/p&gt;
&lt;hr /&gt;
&lt;div&gt;==Creating a Private/Public Key Pair==&lt;br /&gt;
&lt;br /&gt;
If you have never use public key authentication before, you will have to start by creating a key pair. Once the key pair is created, you need to keep the private key in a safe location. Never upload the private key, and never send it by Email. Whoever has your private key can use it to try logging in to ARROW. So, treat the private key as a secret at all times.&lt;br /&gt;
&lt;br /&gt;
The easiest way to prepare a key pair is the use of &amp;quot;puttygen&amp;quot;, part of the &amp;quot;putty&amp;quot; software on Windows. Pressing the Windows Key together with R will open a window where you enter &amp;quot;puttygen&amp;quot;. Once you press &amp;quot;Ok&amp;quot;, the software opens up:&lt;br /&gt;
&lt;br /&gt;
[[File:Puttygen1.png]]&lt;br /&gt;
&lt;br /&gt;
We can go with the defaults and just click on the &amp;quot;Generate&amp;quot; button. Some randomness depends on you moving the mouse around on the screen. Once enough random data is gathered, the private and public key will be generated.&lt;br /&gt;
&lt;br /&gt;
[[File:Puttygen2.png]]&lt;br /&gt;
&lt;br /&gt;
Once the keys have been generated, you need to store the private key in a easy to find location on your local file system. I recommend creating a folder &amp;quot;Keys&amp;quot; within your &amp;quot;Documents&amp;quot; folder, but any other location is fine as well. The idea is that you may easily forget where the key is stored when you need it a later point in time. &lt;br /&gt;
&lt;br /&gt;
[[File:Puttygen3.png]]&lt;br /&gt;
&lt;br /&gt;
Click on &amp;quot;Save private key&amp;quot;, which will create the following dialog box for you:&lt;br /&gt;
&lt;br /&gt;
[[File:Puttygen4.png]]&lt;br /&gt;
&lt;br /&gt;
In an ideal world, assigning a password for your private key would be best, but that also means that you may have to enter that password every time you use the key, which can be a problem. Not assigning a password is one more reason to never risk losing this private key file.&lt;br /&gt;
&lt;br /&gt;
Navigate to the &amp;quot;Keys&amp;quot; folder in the &amp;quot;Documents: folder, and give the key a meaningful name. Something that tells you what it is used for, like &amp;quot;arrow-private&amp;quot;. The extension ppk is automatically added when the key is saved.&lt;br /&gt;
&lt;br /&gt;
Regarding the public key: It is not necessary to save the public key. When loading a private key, the public key is automatically displayed again at the top of the &amp;quot;puttygen&amp;quot; window. It looks somewhat like this:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
ssh-rsa AAAAB3NzaC1yc2EAfAADAQABAAABAQDOHgbwUYAAo+BMpvskvLw6eTqqs8OVUMSbAIobbC2/518PpJ/b&lt;br /&gt;
jW0n9NM4jlNJlEpuT79ZSv5RwSW8rP82qtcLdSEisxhXkyRLLjbzRWQSGxlVUI+K6MfZeFRYD+nqEH4Q1nSvxWJM&lt;br /&gt;
FigswLuWsygcJnGxv0FBGBP7pUrK5pz6AEv5Blf0MCuy0bvuIGhOYEMO4JeId6WYrAwJmZ3mjHqVwO3uQtrYP1Gi&lt;br /&gt;
71A+yk4x2hZepwW5fHiTbeCU/kDycqWPsXOI3e+eBIDBHmaaD7uOkIWT/GnSBTvYr4NkF2xVxnmCsQ5eks13QdY6&lt;br /&gt;
MX1/gjwLl3nOPl6HBLc+KiHPLlMB rsa-key-20260325&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
You can use your mouse to highlight the entire string (starting at ssh-rsa and ending with the date) and copy it into your copy and paste buffer (this is actually a single line altogether). But first we want to do the following:&lt;br /&gt;
&lt;br /&gt;
* create a terminal connection to one of the login nodes (putty, or even ThinLinc)&lt;br /&gt;
* navigate to the folder &amp;quot;~/.ssh&amp;quot;&lt;br /&gt;
* edit or create a file called &amp;quot;authorized_keys&amp;quot;&lt;br /&gt;
* paste the key from puttygen into this file (it should be a single line of text)&lt;br /&gt;
* save the file&lt;/div&gt;</summary>
		<author><name>Ley</name></author>
	</entry>
	<entry>
		<id>https://wiki.anl.gov/wiki_tracc/index.php?title=Private/Public_Key_Authentication&amp;diff=2510</id>
		<title>Private/Public Key Authentication</title>
		<link rel="alternate" type="text/html" href="https://wiki.anl.gov/wiki_tracc/index.php?title=Private/Public_Key_Authentication&amp;diff=2510"/>
		<updated>2026-03-25T19:52:24Z</updated>

		<summary type="html">&lt;p&gt;Ley: /* Creating a Private/Public Key Pair */&lt;/p&gt;
&lt;hr /&gt;
&lt;div&gt;==Creating a Private/Public Key Pair==&lt;br /&gt;
&lt;br /&gt;
If you have never use public key authentication before, you will have to start by creating a key pair. Once the key pair is created, you need to keep the private key in a safe location. Never upload the private key, and never send it by Email. Whoever has your private key can use it to try logging in to ARROW. So, treat the private key as a secret at all times.&lt;br /&gt;
&lt;br /&gt;
The easiest way to prepare a key pair is the use of &amp;quot;puttygen&amp;quot;, part of the &amp;quot;putty&amp;quot; software on Windows. Pressing the Windows Key together with R will open a window where you enter &amp;quot;puttygen&amp;quot;. Once you press &amp;quot;Ok&amp;quot;, the software opens up:&lt;br /&gt;
&lt;br /&gt;
[[File:Puttygen1.png]]&lt;br /&gt;
&lt;br /&gt;
We can go with the defaults and just click on the &amp;quot;Generate&amp;quot; button. Some randomness depends on you moving the mouse around on the screen. Once enough random data is gathered, the private and public key will be generated.&lt;br /&gt;
&lt;br /&gt;
[[File:Puttygen2.png]]&lt;br /&gt;
&lt;br /&gt;
Once the keys have been generated, you need to store the private key in a easy to find location on your local file system. I recommend creating a folder &amp;quot;Keys&amp;quot; within your &amp;quot;Documents&amp;quot; folder, but any other location is fine as well. The idea is that you may easily forget where the key is stored when you need it a later point in time. &lt;br /&gt;
&lt;br /&gt;
[[File:Puttygen3.png]]&lt;br /&gt;
&lt;br /&gt;
Click on &amp;quot;Save private key&amp;quot;, which will create the following dialog box for you:&lt;br /&gt;
&lt;br /&gt;
[[File:Puttygen4.png]]&lt;br /&gt;
&lt;br /&gt;
In an ideal world, assigning a password for your private key would be best, but that also means that you may have to enter that password every time you use the key, which can be a problem. Not assigning a password is one more reason to never risk losing this private key file.&lt;br /&gt;
&lt;br /&gt;
Navigate to the &amp;quot;Keys&amp;quot; folder in the &amp;quot;Documents: folder, and give the key a meaningful name. Something that tells you what it is used for, like &amp;quot;arrow-private&amp;quot;. The extension ppk is automatically added when the key is saved.&lt;br /&gt;
&lt;br /&gt;
Regarding the public key: It is not necessary to save the public key. When loading a private key, the public key is automatically displayed again at the top of the &amp;quot;puttygen&amp;quot; window. It looks somewhat like this:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
ssh-rsa AAAAB3NzaC1yc2EAfAADAQABAAABAQDOHgbwUYAAo+BMpvskvLw6eTqqs8OVUMSbAIobbC2/518PpJ/b&lt;br /&gt;
jW0n9NM4jlNJlEpuT79ZSv5RwSW8rP82qtcLdSEisxhXkyRLLjbzRWQSGxlVUI+K6MfZeFRYD+nqEH4Q1nSvxWJM&lt;br /&gt;
FigswLuWsygcJnGxv0FBGBP7pUrK5pz6AEv5Blf0MCuy0bvuIGhOYEMO4JeId6WYrAwJmZ3mjHqVwO3uQtrYP1Gi&lt;br /&gt;
71A+yk4x2hZepwW5fHiTbeCU/kDycqWPsXOI3e+eBIDBHmaaD7uOkIWT/GnSBTvYr4NkF2xVxnmCsQ5eks13QdY6&lt;br /&gt;
MX1/gjwLl3nOPl6HBLc+KiHPLlMB rsa-key-20260325&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;/div&gt;</summary>
		<author><name>Ley</name></author>
	</entry>
	<entry>
		<id>https://wiki.anl.gov/wiki_tracc/index.php?title=Private/Public_Key_Authentication&amp;diff=2509</id>
		<title>Private/Public Key Authentication</title>
		<link rel="alternate" type="text/html" href="https://wiki.anl.gov/wiki_tracc/index.php?title=Private/Public_Key_Authentication&amp;diff=2509"/>
		<updated>2026-03-25T19:51:13Z</updated>

		<summary type="html">&lt;p&gt;Ley: &lt;/p&gt;
&lt;hr /&gt;
&lt;div&gt;==Creating a Private/Public Key Pair==&lt;br /&gt;
&lt;br /&gt;
If you have never use public key authentication before, you will have to start by creating a key pair. Once the key pair is created, you need to keep the private key in a safe location. Never upload the private key, and never send it by Email. Whoever has your private key can use it to try logging in to ARROW. So, treat the private key as a secret at all times.&lt;br /&gt;
&lt;br /&gt;
The easiest way to prepare a key pair is the use of &amp;quot;puttygen&amp;quot;, part of the &amp;quot;putty&amp;quot; software on Windows. Pressing the Windows Key together with R will open a window where you enter &amp;quot;puttygen&amp;quot;. Once you press &amp;quot;Ok&amp;quot;, the software opens up:&lt;br /&gt;
&lt;br /&gt;
[[File:Puttygen1.png]]&lt;br /&gt;
&lt;br /&gt;
We can go with the defaults and just click on the &amp;quot;Generate&amp;quot; button. Some randomness depends on you moving the mouse around on the screen. Once enough random data is gathered, the private and public key will be generated.&lt;br /&gt;
&lt;br /&gt;
[[File:Puttygen2.png]]&lt;br /&gt;
&lt;br /&gt;
Once the keys have been generated, you need to store the private key in a easy to find location on your local file system. I recommend creating a folder &amp;quot;Keys&amp;quot; within your &amp;quot;Documents&amp;quot; folder, but any other location is fine as well. The idea is that you may easily forget where the key is stored when you need it a later point in time. &lt;br /&gt;
&lt;br /&gt;
[[File:Puttygen3.png]]&lt;br /&gt;
&lt;br /&gt;
Click on &amp;quot;Save private key&amp;quot;, which will create the following dialog box for you:&lt;br /&gt;
&lt;br /&gt;
[[File:Puttygen4.png]]&lt;br /&gt;
&lt;br /&gt;
In an ideal world, assigning a password for your private key would be best, but that also means that you may have to enter that password every time you use the key, which can be a problem. Not assigning a password is one more reason to never risk losing this private key file.&lt;br /&gt;
&lt;br /&gt;
Navigate to the &amp;quot;Keys&amp;quot; folder in the &amp;quot;Documents: folder, and give the key a meaningful name. Something that tells you what it is used for, like &amp;quot;arrow-private&amp;quot;. The extension ppk is automatically added when the key is saved.&lt;br /&gt;
&lt;br /&gt;
Regarding the public key: It is not necessary to save the public key. When loading a private key, the public key is automatically displayed again at the top of the &amp;quot;puttygen&amp;quot; window. It looks somewhat like this:&lt;br /&gt;
&lt;br /&gt;
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDOHgbwUYAAo+BMpvskvLw6eTqqs8OVUMSbAIobbC2/518PpJ/bjW0n9NM4jlNJlEpuT79ZSv5RwSW8rP82qtcLdSEisxhXkyRLLjbzRWQSGxlVUI+K6MfZeFRYD+nqEH4Q1nSvxWJMFigswLuWsygcJnGxv0FBGBP7pUrK5pz6AEv5Blf0MCuy0bvuIGhOYEMO4JeId6WYrAwJmZ3mjHqVwO3uQtrYP1Gi71A+yk4x2hZepwW5fHiTbeCU/kDycqWPsXOI3e+eBIDBHmaaD7uOmIWT/GnSBTvYr4NkF2xVxnmCsQ5eks13QdY6MX1/gjwLl3nOPl6HBLc+KiHPLlMB rsa-key-20260325&lt;/div&gt;</summary>
		<author><name>Ley</name></author>
	</entry>
	<entry>
		<id>https://wiki.anl.gov/wiki_tracc/index.php?title=Private/Public_Key_Authentication&amp;diff=2508</id>
		<title>Private/Public Key Authentication</title>
		<link rel="alternate" type="text/html" href="https://wiki.anl.gov/wiki_tracc/index.php?title=Private/Public_Key_Authentication&amp;diff=2508"/>
		<updated>2026-03-25T19:45:16Z</updated>

		<summary type="html">&lt;p&gt;Ley: /* Creating a Private/Public Key Pair */&lt;/p&gt;
&lt;hr /&gt;
&lt;div&gt;==Creating a Private/Public Key Pair==&lt;br /&gt;
&lt;br /&gt;
If you have never use public key authentication before, you will have to start by creating a key pair. Once the key pair is created, you need to keep the private key in a safe location. Never upload the private key, and never send it by Email. Whoever has your private key can use it to try logging in to ARROW. So, treat the private key as a secret at all times.&lt;br /&gt;
&lt;br /&gt;
The easiest way to prepare a key pair is the use of &amp;quot;puttygen&amp;quot;, part of the &amp;quot;putty&amp;quot; software on Windows. Pressing the Windows Key together with R will open a window where you enter &amp;quot;puttygen&amp;quot;. Once you press &amp;quot;Ok&amp;quot;, the software opens up:&lt;br /&gt;
&lt;br /&gt;
[[File:Puttygen1.png]]&lt;br /&gt;
&lt;br /&gt;
We can go with the defaults and just click on the &amp;quot;Generate&amp;quot; button. Some randomness depends on you moving the mouse around on the screen. Once enough random data is gathered, the private and public key will be generated.&lt;br /&gt;
&lt;br /&gt;
[[File:Puttygen2.png]]&lt;br /&gt;
&lt;br /&gt;
Once the keys have been generated, you need to store the private key in a easy to find location on your local file system. I recommend creating a folder &amp;quot;Keys&amp;quot; within your &amp;quot;Documents&amp;quot; folder, but any other location is fine as well. The idea is that you may easily forget where the key is stored when you need it a later point in time. Click on &amp;quot;Save private key&amp;quot;, and navigate to the &amp;quot;Keys&amp;quot; folder you created.&lt;br /&gt;
&lt;br /&gt;
[[File:Puttygen3.png]]&lt;br /&gt;
&lt;br /&gt;
[[File:Puttygen4.png]]&lt;/div&gt;</summary>
		<author><name>Ley</name></author>
	</entry>
	<entry>
		<id>https://wiki.anl.gov/wiki_tracc/index.php?title=Private/Public_Key_Authentication&amp;diff=2507</id>
		<title>Private/Public Key Authentication</title>
		<link rel="alternate" type="text/html" href="https://wiki.anl.gov/wiki_tracc/index.php?title=Private/Public_Key_Authentication&amp;diff=2507"/>
		<updated>2026-03-25T19:43:30Z</updated>

		<summary type="html">&lt;p&gt;Ley: /* Creating a Private/Public Key Pair */&lt;/p&gt;
&lt;hr /&gt;
&lt;div&gt;==Creating a Private/Public Key Pair==&lt;br /&gt;
&lt;br /&gt;
If you have never use public key authentication before, you will have to start by creating a key pair. Once the key pair is created, you need to keep the private key in a safe location. Never upload the private key, and never send it by Email. Whoever has your private key can use it to try logging in to ARROW. So, treat the private key as a secret at all times.&lt;br /&gt;
&lt;br /&gt;
The easiest way to prepare a key pair is the use of &amp;quot;puttygen&amp;quot;, part of the &amp;quot;putty&amp;quot; software on Windows. Pressing the Windows Key together with R will open a window where you enter &amp;quot;puttygen&amp;quot;. Once you press &amp;quot;Ok&amp;quot;, the software opens up:&lt;br /&gt;
&lt;br /&gt;
[[File:Puttygen1.png]]&lt;br /&gt;
&lt;br /&gt;
We can go with the defaults and just click on the &amp;quot;Generate&amp;quot; button. Some randomness depends on you moving the mouse around on the screen. Once enough random data is gathered, the private and public key will be generated.&lt;br /&gt;
&lt;br /&gt;
[[File:Puttygen2.png]]&lt;br /&gt;
&lt;br /&gt;
You need to store the private key in a easy to find location on your local file system. I recommend creating a folder &amp;quot;Keys&amp;quot; within your &amp;quot;Documents&amp;quot; folder, but any other location is fine as well. The idea is that you may easily forget where the key is stored when you need it a later point in time. Click on &amp;quot;&lt;br /&gt;
&lt;br /&gt;
[[File:Puttygen3.png]]&lt;br /&gt;
&lt;br /&gt;
[[File:Puttygen4.png]]&lt;/div&gt;</summary>
		<author><name>Ley</name></author>
	</entry>
	<entry>
		<id>https://wiki.anl.gov/wiki_tracc/index.php?title=Private/Public_Key_Authentication&amp;diff=2506</id>
		<title>Private/Public Key Authentication</title>
		<link rel="alternate" type="text/html" href="https://wiki.anl.gov/wiki_tracc/index.php?title=Private/Public_Key_Authentication&amp;diff=2506"/>
		<updated>2026-03-25T19:39:59Z</updated>

		<summary type="html">&lt;p&gt;Ley: /* Creating a Private/Public Key Pair */&lt;/p&gt;
&lt;hr /&gt;
&lt;div&gt;==Creating a Private/Public Key Pair==&lt;br /&gt;
&lt;br /&gt;
If you have never use public key authentication before, you will have to start by creating a key pair. Once the key pair is created, you need to keep the private key in a safe location. Never upload the private key, and never send it by Email. Whoever has your private key can use it to try logging in to ARROW. So, treat the private key as a secret at all times.&lt;br /&gt;
&lt;br /&gt;
The easiest way to prepare a key pair is the use of &amp;quot;puttygen&amp;quot;, part of the &amp;quot;putty&amp;quot; software on Windows. Pressing the Windows Key together with R will open a window where you enter &amp;quot;puttygen&amp;quot;. Once you press &amp;quot;Ok&amp;quot;, the software opens up:&amp;lt;/P&amp;gt;&lt;br /&gt;
&lt;br /&gt;
[[File:Puttygen1.png]]&lt;br /&gt;
&lt;br /&gt;
We can go with the defaults and just click on the &amp;quot;Generate&amp;quot; button.&amp;lt;/P&amp;gt;&lt;br /&gt;
&lt;br /&gt;
[[File:Puttygen2.png]]&lt;br /&gt;
&lt;br /&gt;
[[File:Puttygen3.png]]&lt;br /&gt;
&lt;br /&gt;
[[File:Puttygen4.png]]&lt;/div&gt;</summary>
		<author><name>Ley</name></author>
	</entry>
	<entry>
		<id>https://wiki.anl.gov/wiki_tracc/index.php?title=Private/Public_Key_Authentication&amp;diff=2505</id>
		<title>Private/Public Key Authentication</title>
		<link rel="alternate" type="text/html" href="https://wiki.anl.gov/wiki_tracc/index.php?title=Private/Public_Key_Authentication&amp;diff=2505"/>
		<updated>2026-03-25T19:39:30Z</updated>

		<summary type="html">&lt;p&gt;Ley: /* Creating a Private/Public Key Pair */&lt;/p&gt;
&lt;hr /&gt;
&lt;div&gt;==Creating a Private/Public Key Pair==&lt;br /&gt;
&lt;br /&gt;
If you have never use public key authentication before, you will have to start by creating a key pair. Once the key pair is created, you need to keep the private key in a safe location. Never upload the private key, and never send it by Email. Whoever has your private key can use it to try logging in to ARROW. So, treat the private key as a secret at all times.&lt;br /&gt;
&lt;br /&gt;
The easiest way to prepare a key pair is the use of &amp;quot;puttygen&amp;quot;, part of the &amp;quot;putty&amp;quot; software on Windows. Pressing the Windows Key together with R will open a window where you enter &amp;quot;puttygen&amp;quot;. Once you press &amp;quot;Ok&amp;quot;, the software opens up:&lt;br /&gt;
&lt;br /&gt;
[[File:Puttygen1.png]]&lt;br /&gt;
&lt;br /&gt;
We can go with the defaults and just click on the &amp;quot;Generate&amp;quot; button.&amp;lt;P&amp;gt;&lt;br /&gt;
&lt;br /&gt;
[[File:Puttygen2.png]]&lt;br /&gt;
&lt;br /&gt;
[[File:Puttygen3.png]]&lt;br /&gt;
&lt;br /&gt;
[[File:Puttygen4.png]]&lt;/div&gt;</summary>
		<author><name>Ley</name></author>
	</entry>
	<entry>
		<id>https://wiki.anl.gov/wiki_tracc/index.php?title=Private/Public_Key_Authentication&amp;diff=2504</id>
		<title>Private/Public Key Authentication</title>
		<link rel="alternate" type="text/html" href="https://wiki.anl.gov/wiki_tracc/index.php?title=Private/Public_Key_Authentication&amp;diff=2504"/>
		<updated>2026-03-25T19:38:36Z</updated>

		<summary type="html">&lt;p&gt;Ley: /* Creating a Private/Public Key Pair */&lt;/p&gt;
&lt;hr /&gt;
&lt;div&gt;==Creating a Private/Public Key Pair==&lt;br /&gt;
&lt;br /&gt;
If you have never use public key authentication before, you will have to start by creating a key pair. Once the key pair is created, you need to keep the private key in a safe location. Never upload the private key, and never send it by Email. Whoever has your private key can use it to try logging in to ARROW. So, treat the private key as a secret at all times.&lt;br /&gt;
&lt;br /&gt;
The easiest way to prepare a key pair is the use of &amp;quot;puttygen&amp;quot;, part of the &amp;quot;putty&amp;quot; software on Windows. Pressing the Windows Key together with R will open a window where you enter &amp;quot;puttygen&amp;quot;. Once you press &amp;quot;Ok&amp;quot;, the software opens up:&lt;br /&gt;
&lt;br /&gt;
[[File:Puttygen1.png]]&lt;br /&gt;
&lt;br /&gt;
We can go with the defaults and just click on the &amp;quot;Generate&amp;quot; button.&lt;br /&gt;
&lt;br /&gt;
[[File:Puttygen2.png]]&lt;br /&gt;
&lt;br /&gt;
[[File:Puttygen3.png]]&lt;br /&gt;
&lt;br /&gt;
[[File:Puttygen4.png]]&lt;/div&gt;</summary>
		<author><name>Ley</name></author>
	</entry>
	<entry>
		<id>https://wiki.anl.gov/wiki_tracc/index.php?title=File:Puttygen4.png&amp;diff=2503</id>
		<title>File:Puttygen4.png</title>
		<link rel="alternate" type="text/html" href="https://wiki.anl.gov/wiki_tracc/index.php?title=File:Puttygen4.png&amp;diff=2503"/>
		<updated>2026-03-25T19:36:30Z</updated>

		<summary type="html">&lt;p&gt;Ley: &lt;/p&gt;
&lt;hr /&gt;
&lt;div&gt;&lt;/div&gt;</summary>
		<author><name>Ley</name></author>
	</entry>
	<entry>
		<id>https://wiki.anl.gov/wiki_tracc/index.php?title=File:Puttygen3.png&amp;diff=2502</id>
		<title>File:Puttygen3.png</title>
		<link rel="alternate" type="text/html" href="https://wiki.anl.gov/wiki_tracc/index.php?title=File:Puttygen3.png&amp;diff=2502"/>
		<updated>2026-03-25T19:36:06Z</updated>

		<summary type="html">&lt;p&gt;Ley: &lt;/p&gt;
&lt;hr /&gt;
&lt;div&gt;&lt;/div&gt;</summary>
		<author><name>Ley</name></author>
	</entry>
	<entry>
		<id>https://wiki.anl.gov/wiki_tracc/index.php?title=File:Puttygen2.png&amp;diff=2501</id>
		<title>File:Puttygen2.png</title>
		<link rel="alternate" type="text/html" href="https://wiki.anl.gov/wiki_tracc/index.php?title=File:Puttygen2.png&amp;diff=2501"/>
		<updated>2026-03-25T19:35:34Z</updated>

		<summary type="html">&lt;p&gt;Ley: &lt;/p&gt;
&lt;hr /&gt;
&lt;div&gt;&lt;/div&gt;</summary>
		<author><name>Ley</name></author>
	</entry>
	<entry>
		<id>https://wiki.anl.gov/wiki_tracc/index.php?title=Private/Public_Key_Authentication&amp;diff=2500</id>
		<title>Private/Public Key Authentication</title>
		<link rel="alternate" type="text/html" href="https://wiki.anl.gov/wiki_tracc/index.php?title=Private/Public_Key_Authentication&amp;diff=2500"/>
		<updated>2026-03-25T19:30:37Z</updated>

		<summary type="html">&lt;p&gt;Ley: /* Creating a Private/Public Key Pair */&lt;/p&gt;
&lt;hr /&gt;
&lt;div&gt;==Creating a Private/Public Key Pair==&lt;br /&gt;
&lt;br /&gt;
If you have never use public key authentication before, you will have to start by creating a key pair. Once the key pair is created, you need to keep the private key in a safe location. Never upload the private key, and never send it by Email. Whoever has your private key can use it to try logging in to ARROW. So, treat the private key as a secret at all times.&lt;br /&gt;
&lt;br /&gt;
The easiest way to prepare a key pair is the use of &amp;quot;puttygen&amp;quot;, part of the &amp;quot;putty&amp;quot; software on Windows. Pressing the Windows Key together with R will open a window where you enter &amp;quot;puttygen&amp;quot;. Once you press &amp;quot;Ok&amp;quot;, the software opens up:&lt;br /&gt;
&lt;br /&gt;
[[File:Puttygen1.png]]&lt;/div&gt;</summary>
		<author><name>Ley</name></author>
	</entry>
	<entry>
		<id>https://wiki.anl.gov/wiki_tracc/index.php?title=Private/Public_Key_Authentication&amp;diff=2499</id>
		<title>Private/Public Key Authentication</title>
		<link rel="alternate" type="text/html" href="https://wiki.anl.gov/wiki_tracc/index.php?title=Private/Public_Key_Authentication&amp;diff=2499"/>
		<updated>2026-03-25T19:30:18Z</updated>

		<summary type="html">&lt;p&gt;Ley: &lt;/p&gt;
&lt;hr /&gt;
&lt;div&gt;==Creating a Private/Public Key Pair==&lt;br /&gt;
&lt;br /&gt;
If you have never use public key authentication before, you will have to start by creating a key pair. Once the key pair is created, you need to keep the private key in a safe location. Never upload the private key, and never send it by Email. Whoever has your private key can use it to try logging in to ARROW. So, treat the private key as a secret at all times.&lt;br /&gt;
&lt;br /&gt;
The easiest way to prepare a key pair is the use of &amp;quot;puttygen&amp;quot;, part of the &amp;quot;putty&amp;quot; software on Windows. Pressing the Windows Key together with R will open a window where you enter &amp;quot;puttygen&amp;quot;. Once you press &amp;quot;Ok&amp;quot;, the software opens up:&lt;br /&gt;
&lt;br /&gt;
[[File:Puttygen1.jpg]]&lt;/div&gt;</summary>
		<author><name>Ley</name></author>
	</entry>
	<entry>
		<id>https://wiki.anl.gov/wiki_tracc/index.php?title=File:Puttygen1.png&amp;diff=2498</id>
		<title>File:Puttygen1.png</title>
		<link rel="alternate" type="text/html" href="https://wiki.anl.gov/wiki_tracc/index.php?title=File:Puttygen1.png&amp;diff=2498"/>
		<updated>2026-03-25T19:29:41Z</updated>

		<summary type="html">&lt;p&gt;Ley: &lt;/p&gt;
&lt;hr /&gt;
&lt;div&gt;&lt;/div&gt;</summary>
		<author><name>Ley</name></author>
	</entry>
	<entry>
		<id>https://wiki.anl.gov/wiki_tracc/index.php?title=Private/Public_Key_Authentication&amp;diff=2497</id>
		<title>Private/Public Key Authentication</title>
		<link rel="alternate" type="text/html" href="https://wiki.anl.gov/wiki_tracc/index.php?title=Private/Public_Key_Authentication&amp;diff=2497"/>
		<updated>2026-03-25T19:26:11Z</updated>

		<summary type="html">&lt;p&gt;Ley: /* Creating a Private/Public Key Pair */&lt;/p&gt;
&lt;hr /&gt;
&lt;div&gt;==Creating a Private/Public Key Pair==&lt;br /&gt;
&lt;br /&gt;
If you have never use public key authentication before, you will have to start by creating a key pair. Once the key pair is created, you need to keep the private key in a safe location. Never upload the private key, and never send it by Email. Whoever has your private key can use it to try logging in to ARROW. So, treat the private key as a secret at all times.&lt;br /&gt;
&lt;br /&gt;
The easiest way to prepare a key pair is the use of &amp;quot;puttygen&amp;quot;, part of the &amp;quot;putty&amp;quot; software on Windows. Pressing the Windows Key together with R will open a window where you enter &amp;quot;puttygen&amp;quot;. Once you press &amp;quot;Ok&amp;quot;, the software opens up:&lt;br /&gt;
&lt;br /&gt;
* image1&lt;/div&gt;</summary>
		<author><name>Ley</name></author>
	</entry>
	<entry>
		<id>https://wiki.anl.gov/wiki_tracc/index.php?title=Private/Public_Key_Authentication&amp;diff=2496</id>
		<title>Private/Public Key Authentication</title>
		<link rel="alternate" type="text/html" href="https://wiki.anl.gov/wiki_tracc/index.php?title=Private/Public_Key_Authentication&amp;diff=2496"/>
		<updated>2026-03-25T19:22:22Z</updated>

		<summary type="html">&lt;p&gt;Ley: Created page with &amp;quot;==Creating a Private/Public Key Pair==&amp;quot;&lt;/p&gt;
&lt;hr /&gt;
&lt;div&gt;==Creating a Private/Public Key Pair==&lt;/div&gt;</summary>
		<author><name>Ley</name></author>
	</entry>
	<entry>
		<id>https://wiki.anl.gov/wiki_tracc/index.php?title=How_to_Connect&amp;diff=2495</id>
		<title>How to Connect</title>
		<link rel="alternate" type="text/html" href="https://wiki.anl.gov/wiki_tracc/index.php?title=How_to_Connect&amp;diff=2495"/>
		<updated>2026-03-25T19:21:23Z</updated>

		<summary type="html">&lt;p&gt;Ley: &lt;/p&gt;
&lt;hr /&gt;
&lt;div&gt;&lt;br /&gt;
You need an SSH client to connect to Arrow. No other connection protocols are supported. There are 3 general purpose login nodes that you can connect to:&lt;br /&gt;
&lt;br /&gt;
*login1.arrow.tracc.anl.gov&lt;br /&gt;
*login2.arrow.tracc.anl.gov&lt;br /&gt;
*login3.arrow.tracc.anl.gov&lt;br /&gt;
&lt;br /&gt;
There is also a web-based ThinLinc GUI server available at:&lt;br /&gt;
&lt;br /&gt;
*webaccess4.arrow.tracc.anl.gov&lt;br /&gt;
&lt;br /&gt;
which can be accessed using Chrome, FireFox, or Edge. To use this interface, please contact us for instructions.&lt;br /&gt;
&lt;br /&gt;
When connecting, please pick a random entry from the above list to evenly distribute users over multiple machines. Each login node is a unique server, so if you established an interactive GUI session at some point in the past, you can reconnect to the same login node where you started that GUI session, and you will be able to continue where you left off. This is similar to a remote desktop connection in Windows, and is convenient when connecting from an unreliable network or if you simply prefer to be have a more permanent desktop available. Please not that the login servers have to restarted from time to time, so there is guarantee that a session stays around for weeks or months at a time.&lt;br /&gt;
&lt;br /&gt;
For most remote collaborators, your account starts with &amp;quot;ac.&amp;quot;. An example would be &amp;quot;ac.username&amp;quot;. These accounts give you access to the ARROW cluster, but to no other resources at Argonne, without exception. If you are an Argonne employee or have an Argonne domain account  for other reasons, you will use that account to log in. In this case, don&#039;t use the &amp;quot;ANL\&amp;quot; domain prefix as you would on some Windows machines. Just enter the &amp;quot;username&amp;quot; portion.&lt;br /&gt;
&lt;br /&gt;
After entering the password, you will need to confirm your identity using the DUO application. Upon enrollment for a cluster account, you will receive an auto-enrollment Email. Follow the instructions, install the DUO app on your phone, and you are ready to login. As soon as you enter your password in the ssh connection, your phone will prompt you to confirm that you are attempting to log in. After confirming your identity, you will be connected.&lt;br /&gt;
&lt;br /&gt;
Passwords expire on a regular basis (every 6 months or so), and you will need to update your password with Argonne Account Services. You can call 630-252-9999 if you need any help or detailed instructions.&lt;br /&gt;
&lt;br /&gt;
* Access to the TRACC cluster requires the use of an SSH client. Free SSH clients are available for Windows, are included or available for MacOS, and all varieties of Linux or UNIX. SSH clients can be terminal applications to work with command lines (e.g. Putty on Windows), or they can be file transfer utilities (WinSCP, FileZilla, MobaXterm). In any case, cluster connections are all using the SSH protocol (including sftp). Typically you would use a variation of the following command, or provide account and password in your SSH application, such as PuTTy.&lt;br /&gt;
&lt;br /&gt;
 ssh username@login1.arrow.tracc.anl.gov&lt;br /&gt;
&lt;br /&gt;
* If you are planning to use a single GUI application on the cluster through a plain SSH connection and you are either connecting from a Mac or Linux desktop machine or you have an X server installed on your Windows machine, you may want to enable X11 forwarding to your client using the &amp;quot;-X&amp;quot; option on the command line. Performance is limited, but the mechanism works with limited bandwidth and resources:&lt;br /&gt;
&lt;br /&gt;
 ssh -X username@login1.arrow.tracc.anl.gov&lt;br /&gt;
&lt;br /&gt;
In some cases, it is necessary to add the &amp;quot;-Y&amp;quot; option, or to replace the &amp;quot;-X&amp;quot; option with &amp;quot;-Y&amp;quot; altogether. It depends on the graphical application you are planning to use.&lt;br /&gt;
&lt;br /&gt;
 ssh -X -Y username@login1.arrow.tracc.anl.gov&lt;br /&gt;
&lt;br /&gt;
* Alternatively, client software like ThinLinc or X2Go can be used to connect, providing a full graphical user interface and an XFCE desktop. We can provide information on how to download and possibly install the software on your system. We are no longer providing the NoMachine remote desktop software on ARROW.&lt;br /&gt;
&lt;br /&gt;
* [[Private/Public Key Authentication]]&lt;/div&gt;</summary>
		<author><name>Ley</name></author>
	</entry>
	<entry>
		<id>https://wiki.anl.gov/wiki_tracc/index.php?title=How_to_Connect&amp;diff=2494</id>
		<title>How to Connect</title>
		<link rel="alternate" type="text/html" href="https://wiki.anl.gov/wiki_tracc/index.php?title=How_to_Connect&amp;diff=2494"/>
		<updated>2026-03-25T19:18:42Z</updated>

		<summary type="html">&lt;p&gt;Ley: &lt;/p&gt;
&lt;hr /&gt;
&lt;div&gt;&lt;br /&gt;
You need an SSH client to connect to Arrow. No other connection protocols are supported. There are 3 general purpose login nodes that you can connect to:&lt;br /&gt;
&lt;br /&gt;
*login1.arrow.tracc.anl.gov&lt;br /&gt;
*login2.arrow.tracc.anl.gov&lt;br /&gt;
*login3.arrow.tracc.anl.gov&lt;br /&gt;
&lt;br /&gt;
There is also a web-based ThinLinc GUI server available at:&lt;br /&gt;
&lt;br /&gt;
*webaccess4.arrow.tracc.anl.gov&lt;br /&gt;
&lt;br /&gt;
which can be accessed using Chrome, FireFox, or Edge. To use this interface, please contact us for instructions.&lt;br /&gt;
&lt;br /&gt;
When connecting, please pick a random entry from the above list to evenly distribute users over multiple machines. Each login node is a unique server, so if you established an interactive GUI session at some point in the past, you can reconnect to the same login node where you started that GUI session, and you will be able to continue where you left off. This is similar to a remote desktop connection in Windows, and is convenient when connecting from an unreliable network or if you simply prefer to be have a more permanent desktop available. Please not that the login servers have to restarted from time to time, so there is guarantee that a session stays around for weeks or months at a time.&lt;br /&gt;
&lt;br /&gt;
For most remote collaborators, your account starts with &amp;quot;ac.&amp;quot;. An example would be &amp;quot;ac.username&amp;quot;. These accounts give you access to the ARROW cluster, but to no other resources at Argonne, without exception. If you are an Argonne employee or have an Argonne domain account  for other reasons, you will use that account to log in. In this case, don&#039;t use the &amp;quot;ANL\&amp;quot; domain prefix as you would on some Windows machines. Just enter the &amp;quot;username&amp;quot; portion.&lt;br /&gt;
&lt;br /&gt;
After entering the password, you will need to confirm your identity using the DUO application. Upon enrollment for a cluster account, you will receive an auto-enrollment Email. Follow the instructions, install the DUO app on your phone, and you are ready to login. As soon as you enter your password in the ssh connection, your phone will prompt you to confirm that you are attempting to log in. After confirming your identity, you will be connected.&lt;br /&gt;
&lt;br /&gt;
Passwords expire on a regular basis (every 6 months or so), and you will need to update your password with Argonne Account Services. You can call 630-252-9999 if you need any help or detailed instructions.&lt;br /&gt;
&lt;br /&gt;
* Access to the TRACC cluster requires the use of an SSH client. Free SSH clients are available for Windows, are included or available for MacOS, and all varieties of Linux or UNIX. SSH clients can be terminal applications to work with command lines (e.g. Putty on Windows), or they can be file transfer utilities (WinSCP, FileZilla, MobaXterm). In any case, cluster connections are all using the SSH protocol (including sftp). Typically you would use a variation of the following command, or provide account and password in your SSH application, such as PuTTy.&lt;br /&gt;
&lt;br /&gt;
 ssh username@login1.arrow.tracc.anl.gov&lt;br /&gt;
&lt;br /&gt;
* If you are planning to use a single GUI application on the cluster through a plain SSH connection and you are either connecting from a Mac or Linux desktop machine or you have an X server installed on your Windows machine, you may want to enable X11 forwarding to your client using the &amp;quot;-X&amp;quot; option on the command line. Performance is limited, but the mechanism works with limited bandwidth and resources:&lt;br /&gt;
&lt;br /&gt;
 ssh -X username@login1.arrow.tracc.anl.gov&lt;br /&gt;
&lt;br /&gt;
In some cases, it is necessary to add the &amp;quot;-Y&amp;quot; option, or to replace the &amp;quot;-X&amp;quot; option with &amp;quot;-Y&amp;quot; altogether. It depends on the graphical application you are planning to use.&lt;br /&gt;
&lt;br /&gt;
 ssh -X -Y username@login1.arrow.tracc.anl.gov&lt;br /&gt;
&lt;br /&gt;
* Alternatively, client software like ThinLinc or X2Go can be used to connect, providing a full graphical user interface and an XFCE desktop. We can provide information on how to download and possibly install the software on your system. We are no longer providing the NoMachine remote desktop software on ARROW.&lt;/div&gt;</summary>
		<author><name>Ley</name></author>
	</entry>
	<entry>
		<id>https://wiki.anl.gov/wiki_tracc/index.php?title=Job_Submission_and_Monitoring&amp;diff=2493</id>
		<title>Job Submission and Monitoring</title>
		<link rel="alternate" type="text/html" href="https://wiki.anl.gov/wiki_tracc/index.php?title=Job_Submission_and_Monitoring&amp;diff=2493"/>
		<updated>2026-02-26T21:38:19Z</updated>

		<summary type="html">&lt;p&gt;Ley: /* Currently Available StarCCM+ Versions */&lt;/p&gt;
&lt;hr /&gt;
&lt;div&gt;{| align=&amp;quot;right&amp;quot;&lt;br /&gt;
| __TOC__&lt;br /&gt;
|}&lt;br /&gt;
==Resource Summary View==&lt;br /&gt;
&lt;br /&gt;
To get started, users can query the overall status of resources on the cluster. The &#039;&#039;&#039;&amp;quot;qsum&amp;quot;&#039;&#039;&#039; script will list all queues and nodes, as well as how many are offline, down, free, or assigned to users. This is a script developed by our team, and may need to be updated if something goes wrong. Please contact us if you experience any problems.&lt;br /&gt;
&lt;br /&gt;
Each queue groups a number of nodes together based on their hardware and software configurations. Nodes can be part of more than one queue, and there are other complex details that we are ignoring here for the purpose of keeping it simple.&lt;br /&gt;
&lt;br /&gt;
===Queues===&lt;br /&gt;
&lt;br /&gt;
Here is a very brief summary of what each of the queues is, and how to use them efficiently:&lt;br /&gt;
&lt;br /&gt;
; a4000: This is a queue that has three machines with 16 cores each; each of these machines is furthermore equipped with three A4000 GPUs. That makes a total of 9 A4000 GPUs available to users. Neither the GPUs nor the processors are particularly powerful these days, but they make for a good software development environment. The machines have 512GB of memory, which makes them a good platform for experimenting with GPU capabilities.&lt;br /&gt;
; a6000: This is a queue that has only one single machine with 64 cores total, and is equipped with four A6000 GPUs. The system can be upgraded to 8 A6000 GPUs if needed. This is a decent GPU machine that can take a solid workload. The machine has 750GB of memory, which makes for a good production platform.&lt;br /&gt;
; amd16: This is a queue with many of our older AMD-based 16-core machines, each of which equipped with 32GB of memory. While individual machines are a bit outdated, they are all interconnected with Infiniband and can provide a solid production workload in multi-node jobs over MPI without blocking the more current (and thus expensive) systems.&lt;br /&gt;
; epyc1/epyc2: These are 2 separate queues with slightly different performance characteristics. Each of the groups is interconnected with Infiniband to provide a platform for large and demanding software packages, such as LS-Dyna and StarCCM+. They have between 256GB and 512GB of memory. Because licenses for these software packages (LS-Dyna and StarCCM+) are very expensive, these applications should use the two epyc queues for making optimum use of limited core licenses available to each package.&lt;br /&gt;
; xeon28: This is a set of intermediate machines with 28 cores and 64GB of memory. They can be used for a variety of purposes, including MPI jobs and single node application software.&lt;br /&gt;
; virtual: This is a set of nodes without MPI capabilities. They are virtual machines with 32GB each. They can be used for higher demand interactive applications that would interfere otherwise with other users on the login node machines. A user would submit interactive jobs to individual virtual machines and thus avoid any significant load on login nodes.&lt;br /&gt;
&lt;br /&gt;
===The Queue Summary Script (qsum)===&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
$ qsum&lt;br /&gt;
=============== a4000 ==========================================================&lt;br /&gt;
Queue: &amp;quot;a4000&amp;quot; / nodes: 3 / down: 0 / offline: 0 / busy: 0 / available: 3&lt;br /&gt;
      AVAILABLE (3): g001, g002, g003&lt;br /&gt;
=============== a6000 ==========================================================&lt;br /&gt;
Queue: &amp;quot;a6000&amp;quot; / nodes: 1 / down: 0 / offline: 0 / busy: 0 / available: 1&lt;br /&gt;
      AVAILABLE (1): lambda01&lt;br /&gt;
=============== amd16 ==========================================================&lt;br /&gt;
Queue: &amp;quot;amd16&amp;quot; / nodes: 98 / down: 28 / offline: 5 / busy: 4 / available: 61&lt;br /&gt;
          DOWN (28): n012, n017, n018, n022, n024, n030, n034, n037, n038, n040&lt;br /&gt;
                     n041, n042, n056, n057, n060, n062, n064, n069, n071, n072&lt;br /&gt;
                     n076, n079, n080, n086, n087, n089, n093, n094&lt;br /&gt;
        OFFLINE (5): n001, n002, n003, n004, n010&lt;br /&gt;
       ac.xliu1 (4): n026, n027, n028, n029&lt;br /&gt;
     AVAILABLE (61): n005, n006, n007, n008, n009, n011, n013, n014, n015, n016&lt;br /&gt;
                     n019, n020, n021, n023, n025, n031, n032, n033, n035, n036&lt;br /&gt;
                     n039, n043, n044, n045, n046, n047, n048, n049, n050, n051&lt;br /&gt;
                     n052, n053, n054, n055, n058, n059, n061, n063, n065, n066&lt;br /&gt;
                     n067, n068, n070, n073, n074, n075, n077, n078, n081, n082&lt;br /&gt;
                     n083, n084, n085, n088, n090, n091, n095, n096, n097, n098&lt;br /&gt;
                     n099&lt;br /&gt;
=============== epyc1 ==========================================================&lt;br /&gt;
Queue: &amp;quot;epyc1&amp;quot; / nodes: 27 / down: 1 / offline: 0 / busy: 10 / available: 16&lt;br /&gt;
           DOWN (1): a011&lt;br /&gt;
       ac.ge.w* (2): a004, a027&lt;br /&gt;
       ac.razm* (2): a015, a016&lt;br /&gt;
       ac.vpra* (4): a005, a006, a008, a009&lt;br /&gt;
         msitek (2): a001, a002&lt;br /&gt;
     AVAILABLE (16): a003, a007, a010, a012, a013, a014, a017, a018, a019, a020&lt;br /&gt;
                     a021, a022, a023, a024, a025, a026&lt;br /&gt;
=============== epyc2 ==========================================================&lt;br /&gt;
Queue: &amp;quot;epyc2&amp;quot; / nodes: 20 / down: 0 / offline: 0 / busy: 16 / available: 4&lt;br /&gt;
      cbojano* (16): a028, a030, a031, a032, a033, a034, a035, a036, a037, a038&lt;br /&gt;
                     a039, a040, a044, a045, a046, a047&lt;br /&gt;
      AVAILABLE (4): a029, a041, a042, a043&lt;br /&gt;
=============== virtual ========================================================&lt;br /&gt;
Queue: &amp;quot;virtual&amp;quot; / nodes: 6 / down: 0 / offline: 0 / busy: 0 / available: 6&lt;br /&gt;
      AVAILABLE (6): v001, v002, v003, v004, v005, v006&lt;br /&gt;
=============== xeon28 =========================================================&lt;br /&gt;
Queue: &amp;quot;xeon28&amp;quot; / nodes: 12 / down: 0 / offline: 0 / busy: 0 / available: 12&lt;br /&gt;
     AVAILABLE (12): p001, p002, p003, p004, p005, p006, p007, p008, p009, p010&lt;br /&gt;
                     p011, p012&lt;br /&gt;
================================================================================&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
==Queue Status and Monitoring Jobs==&lt;br /&gt;
&lt;br /&gt;
===qstat===&lt;br /&gt;
&lt;br /&gt;
To find out out about the status of all running jobs on the cluster you can use the &#039;&#039;&#039;&amp;quot;qstat&amp;quot;&#039;&#039;&#039; command. Here is an example:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
$ qstat&lt;br /&gt;
&lt;br /&gt;
Nov 20 18:30 ley@login3:Plots$ qstat&lt;br /&gt;
Job id            Name             User              Time Use S Queue&lt;br /&gt;
----------------  ---------------- ----------------  -------- - -----&lt;br /&gt;
3023.pbs          STDIN            msitek            4144:14* R epyc2           &lt;br /&gt;
3029.pbs          STDIN            ley               76:46:53 R epyc2           &lt;br /&gt;
3032.pbs          STDIN            msitek            2879:52* R epyc2           &lt;br /&gt;
3033.pbs          STDIN            msitek            3687:29* R epyc2           &lt;br /&gt;
3048.pbs          foo.sh           james.cook               0 Q amd16           &lt;br /&gt;
3060.pbs          of13.sh          ley               310:47:* R epyc2           &lt;br /&gt;
3061.pbs          of13.sh          ley               308:37:* R epyc2           &lt;br /&gt;
3062.pbs          of13.sh          ley               308:02:* R epyc2           &lt;br /&gt;
3063.pbs          of13.sh          ley               308:15:* R epyc2&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
The first column shows the &#039;&#039;&#039;job id&#039;&#039;&#039;, a unique identifier for all jobs ever submitted to the cluster. This job id is important when killing jobs, or for other actions you may need to take.&lt;br /&gt;
&lt;br /&gt;
The next column shows the name of the job script. If the column shows &#039;&#039;&#039;STDIN&#039;&#039;&#039;, it means that this is an &#039;&#039;&#039;interactive job&#039;&#039;&#039; where a user can enter commands in a terminal window. This is particularly useful for model and software development task where the application has to be started and killed repeatedly.&lt;br /&gt;
&lt;br /&gt;
The owner of the job is shown next. These are the user names of the various people using the cluster.&lt;br /&gt;
&lt;br /&gt;
The last three columns indicate the current run time of the job, whether it is running (R) or waiting (Q) for execution. The last entry shows the queue in which the job is running.&lt;br /&gt;
&lt;br /&gt;
===qstat -an1===&lt;br /&gt;
&lt;br /&gt;
Adding a few options gives much more detail about each jobs.&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
qstat -an1&lt;br /&gt;
&lt;br /&gt;
Nov 20 13:09 ley@login3:Plots$ qstat -an1&lt;br /&gt;
&lt;br /&gt;
pbs: &lt;br /&gt;
                                                            Req&#039;d  Req&#039;d   Elap&lt;br /&gt;
Job ID          Username Queue    Jobname    SessID NDS TSK Memory Time  S Time&lt;br /&gt;
--------------- -------- -------- ---------- ------ --- --- ------ ----- - -----&lt;br /&gt;
3023.pbs        msitek   epyc2    STDIN      24360*   1  64  350gb 100:0 R 81:46 a028/0*64&lt;br /&gt;
3029.pbs        ley      epyc2    STDIN      21719*   2 128  100gb 200:0 R 72:31 a030/0*64+a031/0*64&lt;br /&gt;
3032.pbs        msitek   epyc2    STDIN      18102*   1  64  350gb 100:0 R 57:57 a029/0*64&lt;br /&gt;
3033.pbs        msitek   epyc2    STDIN      830486   1  64  350gb 100:0 R 57:53 a032/0*64&lt;br /&gt;
3048.pbs        james.c* amd16    foo.sh        --    1  28   30gb 01:00 Q   --   -- &lt;br /&gt;
3060.pbs        ley      epyc2    STDIN      763101   1  64  350gb 48:00 R 06:42 a033/0*64&lt;br /&gt;
3061.pbs        ley      epyc2    STDIN      763947   1  64  350gb 48:00 R 06:40 a034/0*64&lt;br /&gt;
3062.pbs        ley      epyc2    STDIN      761473   1  64  350gb 48:00 R 06:39 a035/0*64&lt;br /&gt;
3063.pbs        ley      epyc2    STDIN      766205   1  64  350gb 48:00 R 06:40 a036/0*64&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
In this table, you can see how many nodes and cores are being used by each job. For example, job 3029 of the user &amp;quot;ley&amp;quot; shows that it is running on 2 nodes using a total of 128 cores. In addition to the elapsed time, the table also show the reserved time for this job. This allow you to estimate when a job will be definitely finalized (or killed by the system if still running).&lt;br /&gt;
&lt;br /&gt;
The last column (without a header) is written because the option &#039;&#039;&#039;&amp;quot;-an1&amp;quot;&#039;&#039;&#039; was used. This useful to learn about which nodes are used by each job.&lt;br /&gt;
&lt;br /&gt;
===qstat -q===&lt;br /&gt;
&lt;br /&gt;
To learn more about the queues on the cluster, the &#039;&#039;&#039;&amp;quot;-q&amp;quot;&#039;&#039;&#039; option turns out to be useful.&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
$ qstat -q&lt;br /&gt;
&lt;br /&gt;
server: pbs&lt;br /&gt;
&lt;br /&gt;
Queue            Memory CPU Time Walltime Node   Run   Que   Lm  State&lt;br /&gt;
---------------- ------ -------- -------- ---- ----- ----- ----  -----&lt;br /&gt;
virtual            30gb    --       --       1     0     0   --   E R&lt;br /&gt;
a4000             500gb    --       --       1     0     0   --   E R&lt;br /&gt;
a6000             750gb    --       --       1     0     0   --   E R&lt;br /&gt;
xeon28             --      --       --       4     0     0   --   E R&lt;br /&gt;
amd16              --      --       --       8     0     1   --   E R&lt;br /&gt;
epyc2              --      --       --       2    14     0   --   E R&lt;br /&gt;
epyc1              --      --       --       2     0     0   --   E R&lt;br /&gt;
                                               ----- -----&lt;br /&gt;
                                                  14     1&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
For each queue, some basic values are displayed. The first three queues listed above have a default memory allocation as shown, and the column &#039;&#039;&#039;&amp;quot;Node&amp;quot;&#039;&#039;&#039; indicates the maximum number of nodes that can be asked for at job submission time. For example, you can request just one node for a job from the first three queues (because these are queues where MPI makes no sense). The &#039;&#039;&#039;&amp;quot;xeon28&amp;quot;&#039;&#039;&#039; queue also for a maximum of 4 nodes per MPI job. The &#039;&#039;&#039;&amp;quot;amd16&amp;quot;&#039;&#039;&#039; queue has a maximum of 8 nodes per job, and the &#039;&#039;&#039;&amp;quot;epyc1&amp;quot;&#039;&#039;&#039; and &#039;&#039;&#039;&amp;quot;epyc2&amp;quot;&#039;&#039;&#039; queues have maxima of two nodes per job. These limitations can be changed by the administrator as needed. As shown above, this will prevent inefficient resource requests.&lt;br /&gt;
&lt;br /&gt;
===qstat -f===&lt;br /&gt;
&lt;br /&gt;
The command &#039;&#039;&#039;&amp;quot;qstat -f -F json 3029&amp;quot;&#039;&#039;&#039; retrieves extremely detailed stats on the running job 3029. The result can be returned in JSON format to be ready for further processing (shown below).&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
$ qstat -f -F json 3029&lt;br /&gt;
{&lt;br /&gt;
    &amp;quot;timestamp&amp;quot;:1763705353,&lt;br /&gt;
    &amp;quot;pbs_version&amp;quot;:&amp;quot;23.06.06&amp;quot;,&lt;br /&gt;
    &amp;quot;pbs_server&amp;quot;:&amp;quot;pbs&amp;quot;,&lt;br /&gt;
    &amp;quot;Jobs&amp;quot;:{&lt;br /&gt;
        &amp;quot;3029.pbs&amp;quot;:{&lt;br /&gt;
            &amp;quot;Job_Name&amp;quot;:&amp;quot;STDIN&amp;quot;,&lt;br /&gt;
            &amp;quot;Job_Owner&amp;quot;:&amp;quot;ley@login4&amp;quot;,&lt;br /&gt;
            &amp;quot;resources_used&amp;quot;:{&lt;br /&gt;
                &amp;quot;cpupercent&amp;quot;:98,&lt;br /&gt;
                &amp;quot;cput&amp;quot;:&amp;quot;76:46:53&amp;quot;,&lt;br /&gt;
                &amp;quot;hpmem&amp;quot;:&amp;quot;0b&amp;quot;,&lt;br /&gt;
                &amp;quot;mem&amp;quot;:&amp;quot;52428800kb&amp;quot;,&lt;br /&gt;
                &amp;quot;ncpus&amp;quot;:128,&lt;br /&gt;
                &amp;quot;vmem&amp;quot;:&amp;quot;52428800kb&amp;quot;,&lt;br /&gt;
                &amp;quot;walltime&amp;quot;:&amp;quot;78:09:32&amp;quot;&lt;br /&gt;
            },&lt;br /&gt;
            &amp;quot;job_state&amp;quot;:&amp;quot;R&amp;quot;,&lt;br /&gt;
            &amp;quot;queue&amp;quot;:&amp;quot;epyc2&amp;quot;,&lt;br /&gt;
            &amp;quot;server&amp;quot;:&amp;quot;pbs&amp;quot;,&lt;br /&gt;
            &amp;quot;Checkpoint&amp;quot;:&amp;quot;u&amp;quot;,&lt;br /&gt;
            &amp;quot;ctime&amp;quot;:&amp;quot;Mon Nov 17 17:58:25 2025&amp;quot;,&lt;br /&gt;
            &amp;quot;Error_Path&amp;quot;:&amp;quot;/dev/pts/0&amp;quot;,&lt;br /&gt;
            &amp;quot;exec_host&amp;quot;:&amp;quot;a030/0*64+a031/0*64&amp;quot;,&lt;br /&gt;
            &amp;quot;exec_vnode&amp;quot;:&amp;quot;(a030:ncpus=64:mem=52428800kb)+(a031:ncpus=64:mem=52428800kb)&amp;quot;,&lt;br /&gt;
            &amp;quot;Hold_Types&amp;quot;:&amp;quot;n&amp;quot;,&lt;br /&gt;
            &amp;quot;interactive&amp;quot;:&amp;quot;True&amp;quot;,&lt;br /&gt;
            &amp;quot;Join_Path&amp;quot;:&amp;quot;n&amp;quot;,&lt;br /&gt;
            &amp;quot;Keep_Files&amp;quot;:&amp;quot;n&amp;quot;,&lt;br /&gt;
            &amp;quot;Mail_Points&amp;quot;:&amp;quot;a&amp;quot;,&lt;br /&gt;
            &amp;quot;mtime&amp;quot;:&amp;quot;Fri Nov 21 00:07:59 2025&amp;quot;,&lt;br /&gt;
            &amp;quot;Output_Path&amp;quot;:&amp;quot;/dev/pts/0&amp;quot;,&lt;br /&gt;
            &amp;quot;Priority&amp;quot;:0,&lt;br /&gt;
            &amp;quot;qtime&amp;quot;:&amp;quot;Mon Nov 17 17:58:25 2025&amp;quot;,&lt;br /&gt;
            &amp;quot;Rerunable&amp;quot;:&amp;quot;False&amp;quot;,&lt;br /&gt;
            &amp;quot;Resource_List&amp;quot;:{&lt;br /&gt;
                &amp;quot;mem&amp;quot;:&amp;quot;100gb&amp;quot;,&lt;br /&gt;
                &amp;quot;mpiprocs&amp;quot;:128,&lt;br /&gt;
                &amp;quot;ncpus&amp;quot;:128,&lt;br /&gt;
                &amp;quot;nodect&amp;quot;:2,&lt;br /&gt;
                &amp;quot;place&amp;quot;:&amp;quot;free&amp;quot;,&lt;br /&gt;
                &amp;quot;select&amp;quot;:&amp;quot;2:ncpus=64:mem=50gb:mpiprocs=64&amp;quot;,&lt;br /&gt;
                &amp;quot;walltime&amp;quot;:&amp;quot;200:00:00&amp;quot;&lt;br /&gt;
            },&lt;br /&gt;
            &amp;quot;stime&amp;quot;:&amp;quot;Mon Nov 17 17:58:25 2025&amp;quot;,&lt;br /&gt;
            &amp;quot;session_id&amp;quot;:2171964,&lt;br /&gt;
            &amp;quot;jobdir&amp;quot;:&amp;quot;/mnt/lustre/arrow/home/ley&amp;quot;,&lt;br /&gt;
            &amp;quot;substate&amp;quot;:42,&lt;br /&gt;
            &amp;quot;Variable_List&amp;quot;:{&lt;br /&gt;
                &amp;quot;PBS_O_HOME&amp;quot;:&amp;quot;/mnt/lustre/arrow/home/ley&amp;quot;,&lt;br /&gt;
                &amp;quot;PBS_O_LANG&amp;quot;:&amp;quot;en_US.UTF-8&amp;quot;,&lt;br /&gt;
                &amp;quot;PBS_O_LOGNAME&amp;quot;:&amp;quot;ley&amp;quot;,&lt;br /&gt;
                &amp;quot;PBS_O_PATH&amp;quot;:&amp;quot;/shared/apps/active/lstc/lsprepost/SP-4.5:/shared/apps/active/lstc/lsprepost/DP-4.3:/shared/bin:/usr/share/Modules/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/pbs/bin:/opt/thinlinc/bin:/opt/thinlinc/sbin:/mnt/lustre/arrow/home/ley/.local/bin:/mnt/lustre/arrow/home/ley/bin&amp;quot;,&lt;br /&gt;
                &amp;quot;PBS_O_MAIL&amp;quot;:&amp;quot;/var/spool/mail/ley&amp;quot;,&lt;br /&gt;
                &amp;quot;PBS_O_SHELL&amp;quot;:&amp;quot;/bin/bash&amp;quot;,&lt;br /&gt;
                &amp;quot;PBS_O_WORKDIR&amp;quot;:&amp;quot;/mnt/lustre/arrow/home/ley/Qualification/LS-Dyna/Rocky9/Seatbelt/Template&amp;quot;,&lt;br /&gt;
                &amp;quot;PBS_O_SYSTEM&amp;quot;:&amp;quot;Linux&amp;quot;,&lt;br /&gt;
                &amp;quot;PBS_O_QUEUE&amp;quot;:&amp;quot;epyc2&amp;quot;,&lt;br /&gt;
                &amp;quot;PBS_O_HOST&amp;quot;:&amp;quot;login4&amp;quot;&lt;br /&gt;
            },&lt;br /&gt;
            &amp;quot;comment&amp;quot;:&amp;quot;Job run at Mon Nov 17 at 17:58 on (a030:ncpus=64:mem=52428800kb)+(a031:ncpus=64:mem=52428800kb)&amp;quot;,&lt;br /&gt;
            &amp;quot;etime&amp;quot;:&amp;quot;Mon Nov 17 17:58:25 2025&amp;quot;,&lt;br /&gt;
            &amp;quot;run_count&amp;quot;:1,&lt;br /&gt;
            &amp;quot;Submit_arguments&amp;quot;:&amp;quot;-I -q epyc2 -l walltime=200:00:00,select=2:ncpus=64:mem=50gb:mpiprocs=64&amp;quot;,&lt;br /&gt;
            &amp;quot;project&amp;quot;:&amp;quot;_pbs_project_default&amp;quot;,&lt;br /&gt;
            &amp;quot;Submit_Host&amp;quot;:&amp;quot;login4&amp;quot;&lt;br /&gt;
        }&lt;br /&gt;
    }&lt;br /&gt;
}&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
===Manual pages for qstat===&lt;br /&gt;
&lt;br /&gt;
To learn more about the &#039;&#039;&#039;&amp;quot;qstat&amp;quot;&#039;&#039;&#039; command, you can use the command &#039;&#039;&#039;&amp;quot;man qstat&amp;quot;&#039;&#039;&#039;, which will print a lot of detailed information about the capabilities of this command.&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
$ man qstat&lt;br /&gt;
&lt;br /&gt;
qstat(1B)                                       PBS Professional                                       qstat(1B)&lt;br /&gt;
&lt;br /&gt;
NAME&lt;br /&gt;
       qstat - display status of PBS jobs, queues, or servers&lt;br /&gt;
&lt;br /&gt;
SYNOPSIS&lt;br /&gt;
       Displaying Job Status&lt;br /&gt;
              Default format:&lt;br /&gt;
              qstat [-E] [-J] [-p] [-t] [-w] [-x] [[&amp;lt;job ID&amp;gt; | &amp;lt;destination&amp;gt;] ...]&lt;br /&gt;
&lt;br /&gt;
              Long format:&lt;br /&gt;
              qstat -f [-F json | dsv [-D &amp;lt;delimiter&amp;gt;]] [-E] [-J] [-p] [-t] [-w]&lt;br /&gt;
                    [-x] [[&amp;lt;job ID&amp;gt; | &amp;lt;destination&amp;gt;] ...]&lt;br /&gt;
... &amp;lt;many more pages&amp;gt;&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
==Job Submission Basics==&lt;br /&gt;
&lt;br /&gt;
Jobs are submitted into the system using the &#039;&#039;&#039;&amp;quot;qsub&amp;quot;&#039;&#039;&#039; application. This application can take many different options and allows for a lot of different resource requests to tell the cluster what to do. We are running &#039;&#039;&#039;OpenPBS 23.06.06&#039;&#039;&#039; as our job scheduler. Here is a link to the User&#039;s Manual (of PBS PRO) if you want to explore gory details and capabilities. The User&#039;s Guide has about 240 pages, the Reference Guide has 500 pages, and the Big Book has 2500 pages. So there is a lot of information available. I also added job submission info for the LCRC cluster.&lt;br /&gt;
&lt;br /&gt;
* [https://argonne-lcrc.github.io/user-guides/running-jobs-at-lcrc/pbs-pro/ Argonne&#039;s LCRC pages on job submissions on their clusters]&lt;br /&gt;
* [https://help.altair.com/2022.1.0/PBS%20Professional/PBSUserGuide2022.1.pdf PBS Professional 2022.1 User&#039;s Guide]&lt;br /&gt;
* [https://help.altair.com/2022.1.0/PBS%20Professional/PBSReferenceGuide2022.1.pdf PBS Professional 2022.1 Reference Guide]&lt;br /&gt;
* [https://help.altair.com/2022.1.0/PBS%20Professional/PBS2022.1.pdf Altair PBS Professional 2022.1 Big Book]&lt;br /&gt;
&lt;br /&gt;
The User&#039;s Guide can be very helpful to clarify some of the concepts and capabilities, but it can be hard to find the specific information you may be looking for. Please understand that we are no longer running TORQUE and MAUI, so the syntax for job submission is distinctively different yet quite similar.&lt;br /&gt;
&lt;br /&gt;
The reference guide may be helpful to understand the complete syntax and full capabilities of the software.&lt;br /&gt;
&lt;br /&gt;
The big book is what I had to use when configuring OpenPBS earlier this year. This includes all the tricky details needed to make the system work smoothly for us. It&#039;s a bit scary to look at a PDF file that is 2500 pages long, but that is nothing compared to the StarCCM+ manuals.&lt;br /&gt;
&lt;br /&gt;
&amp;lt;BLOCKQUOTE&amp;gt;&lt;br /&gt;
[[File:Attention.jpg|25px]] &#039;&#039;&#039;IMPORTANT NOTE:&#039;&#039;&#039; &#039;&#039;The following sections are important to understand. They explain how jobs are submitted and then scjeduled for execution based on resources available and the specific need of the user.&#039;&#039;&lt;br /&gt;
&amp;lt;/BLOCKQUOTE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
The following sections explain the various tasks you may want to submit fir execution, ordered from simple to complex.&lt;br /&gt;
&lt;br /&gt;
* General Batch Jobs&lt;br /&gt;
** Requesting a Single Node for a Job&lt;br /&gt;
** Requesting Multiple Nodes for a Job&lt;br /&gt;
* Embedded Job Resource Requests&lt;br /&gt;
* Interactive Jobs&lt;br /&gt;
* Interactive Jobs with X-Windows GUI Applications&lt;br /&gt;
* Running Multiple Jobs on Single Nodes&lt;br /&gt;
* Running Jobs using GPUs&lt;br /&gt;
&lt;br /&gt;
===General Batch Jobs===&lt;br /&gt;
&lt;br /&gt;
Let&#039;s get started with a very basic usage of the system. Let&#039;s assume you have a simple application, and you want to execute it on a cluster node. Let&#039;s also assume that this is a very simple application, one that runs on one or a few cores, doesn&#039;t require any keyboard interaction with the user, doesn&#039;t need the user to see what&#039;s typically written to the screen, and writes its output to files. In this case, we can submit this application as a batch job, which will place it into an execution queue and process it as soon as a node becomes available.&lt;br /&gt;
&lt;br /&gt;
If the application requires more cores than a single node can provide, we can run the application over Infiniband with MPI message passing. In this case, we need to understand the concept of MPI applications a bit better. In both cases, we get started by creating a folder on the file system. Naming conventions are important so that you can distinguish the jobs by folder name.&lt;br /&gt;
&lt;br /&gt;
For both of the above scenarios, you would typically create a Bash shell script, and then submit the script into one of the queues for eventual execution.&lt;br /&gt;
&lt;br /&gt;
====Requesting a Single Node for a Job====&lt;br /&gt;
&lt;br /&gt;
Let&#039;s try something rather trivial to get used to the concept. Create yourself a folder, for example &#039;&#039;&#039;&amp;quot;myjobfolder&amp;quot;&#039;&#039;&#039;. Within that folder, create a job submission script. That script can have any name, but something short and simple may be best. Let&#039;s assume you create a file called &#039;&#039;&#039;&amp;quot;cluster.job&amp;quot;&#039;&#039;&#039;. The file doesn&#039;t have to have that extension. Any file name will do. But using the same filename for all of your jobs helps finding your way around the many files that will be created over time. The &#039;&#039;&#039;&amp;quot;cluster.job&amp;quot;&#039;&#039;&#039; file should look something like this:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
#!/bin/bash&lt;br /&gt;
#&lt;br /&gt;
# the following ensures that you will change into the directory where you are&lt;br /&gt;
# submitting the job from.&lt;br /&gt;
cd $PBS_O_WORKDIR&lt;br /&gt;
#&lt;br /&gt;
# now we sleep for 60 seconds and waste time. This is a placeholder for your application,&lt;br /&gt;
# which would be doing useful work for you.&lt;br /&gt;
sleep 60&lt;br /&gt;
#&lt;br /&gt;
# and after doing things, we may want to write something into a file to show that&lt;br /&gt;
# our jobs is done.&lt;br /&gt;
echo `date` &amp;gt; info.log&lt;br /&gt;
#&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
This can be submitted without detailed resource specifications (except for the walltime, which is be default 0:00:00):&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
$ qsub -q virtual -l walltime=1:00:00 cluster.job &lt;br /&gt;
3072.pbs&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
Wait a little, then check the status of running jobs:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
$ qstat -an1&lt;br /&gt;
&lt;br /&gt;
pbs: &lt;br /&gt;
                                                            Req&#039;d  Req&#039;d   Elap&lt;br /&gt;
Job ID          Username Queue    Jobname    SessID NDS TSK Memory Time  S Time&lt;br /&gt;
--------------- -------- -------- ---------- ------ --- --- ------ ----- - -----&lt;br /&gt;
3023.pbs        msitek   epyc2    STDIN      24360*   1  64  350gb 100:0 R 83:17 a028/0*64&lt;br /&gt;
3029.pbs        ley      epyc2    STDIN      21719*   2 128  100gb 200:0 R 74:00 a030/0*64+a031/0*64&lt;br /&gt;
3033.pbs        msitek   epyc2    STDIN      830486   1  64  350gb 100:0 R 59:23 a032/0*64&lt;br /&gt;
3048.pbs        james.c* amd16    foo.sh        --    1  28   30gb 01:00 Q   --   -- &lt;br /&gt;
3060.pbs        ley      epyc2    STDIN      763101   1  64  350gb 48:00 R 08:10 a033/0*64&lt;br /&gt;
3061.pbs        ley      epyc2    STDIN      763947   1  64  350gb 48:00 R 08:10 a034/0*64&lt;br /&gt;
3070.pbs        ley      epyc2    STDIN      766847   1  64  350gb 48:00 R 07:23 a042/0*64&lt;br /&gt;
3072.pbs        ley      virtual  cluster.j* 230230   1   4   30gb 01:00 E 00:01 v001/0*4&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
In this particular example, we are sending this job to the &#039;&#039;&#039;queue &amp;quot;virtual&amp;quot;&#039;&#039;&#039;. This queue, by default, allocates 30GB of memory to the job, and runs on 1 node with 4 cores. This is sufficient capacity to run quite a workload. When submitting a job to a single node, &#039;&#039;&#039;reasonable maximum allocations are automatically assigned&#039;&#039;&#039;, and the user doesn&#039;t have to worry about running out of memory or how many cores he will be using.&lt;br /&gt;
&lt;br /&gt;
The only required argument is the &#039;&#039;&#039;&amp;quot;walltime&amp;quot;&#039;&#039;&#039; argument. By default, the job will quit as soon as it is submitted. This indicates to the user that he forgot to provide the &#039;&#039;&#039;&amp;quot;walltime&amp;quot;&#039;&#039;&#039; argument.&lt;br /&gt;
&lt;br /&gt;
When the job disappears from the job list, it is done. At this point, you will find the file &amp;quot;info.log&amp;quot; in your job folder.&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
$ cat info.log &lt;br /&gt;
Thu Nov 20 08:00:31 PM CST 2025&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
====Requesting Multiple Nodes for a Job====&lt;br /&gt;
&lt;br /&gt;
To run jobs on multiple nodes, you will be likely &#039;&#039;&#039;executing jobs using MPI&#039;&#039;&#039;, the message passing interface. This establishes high-speed low-latency interconnections between the cores on one machine and the cores on the other machines. Data transfer does not require involvement of the cores themselves. Instead, the core tell the InfiniBand interconnect (and cores on the same node through shared memory) to transfer the data through RDMA, remoted direct memory access. The cores don&#039;t need to spend CPU cycles on copying data, but rather simply access the data once it has been copied by the Infiniband fabric. This makes for extremely efficient remote memory access, and message passing is used to coordinate data transfer between the cores no matter where they are located on any of the nodes.&lt;br /&gt;
&lt;br /&gt;
On our cluster, MPI-aware applications like &#039;&#039;&#039;OpenFOAM&#039;&#039;&#039;, &#039;&#039;&#039;StarCCM+&#039;&#039;&#039;, and &#039;&#039;&#039;LS-Dyna&#039;&#039;&#039; can be loaded as modules, which then automatically selects the most appropriate MPI library to use. The software applications have been tested to ensure that they work out-of-the box if a user selects any specific version of any of the applications.&lt;br /&gt;
&lt;br /&gt;
The following is a very trivial example for the MPI execution of a very simple executable, with one copy running on each core of the nodes allocated to  the job. It doesn&#039;t perform any real work and just wastes resources for a short time, but it illustrates how execution on the cores of various nodes works.&lt;br /&gt;
&lt;br /&gt;
Like in the previous section, we start with a simple job script that we submit to an appropriate queues. In this case, we pick a queue that has machines with Infiniband interfaces supporting efficient communications. Let&#039;s assume we edit a file with the name &#039;&#039;&#039;&amp;quot;parallel.job&amp;quot;&#039;&#039;&#039; like this:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
#!/bin/bash&lt;br /&gt;
#&lt;br /&gt;
# the following ensures that you will change into the directory where you are&lt;br /&gt;
# submitting the job from.&lt;br /&gt;
cd $PBS_O_WORKDIR&lt;br /&gt;
#&lt;br /&gt;
# to execute a simple command on all of the cores of all of the nodes allocated to the job,&lt;br /&gt;
# we need to make one of the MPI versions available. Let&#039;s use one of the most up-to-date&lt;br /&gt;
# MPI library available on the cluster&lt;br /&gt;
module load intel/2024.2.0/mpi/2021.13&lt;br /&gt;
#&lt;br /&gt;
# now we are apply a few settings that ensure that the MPI library will use the highest-performing&lt;br /&gt;
# Infiniband Interconnect, as well as a few options to tell MPI how to interface nodes with&lt;br /&gt;
# each other and which specific Infiniband adapter to use. This is complex and requires in-depth&lt;br /&gt;
# knowledge of the QLogic Infiniband adapters we are using. It is unlikely that you will ever have to&lt;br /&gt;
# deal with these options, because the &amp;quot;module load&amp;quot; command for the engineering applications we provide&lt;br /&gt;
# on ARROW will handle all those details transparently without the user needing to understand the details.&lt;br /&gt;
export I_MPI_HYDRA_BOOTSTRAP=ssh&lt;br /&gt;
export MPI_DEVICE=rdma:ofa-v2-ib0&lt;br /&gt;
export UCX_NET_DEVICES=qib0:1&lt;br /&gt;
#&lt;br /&gt;
# it doesn&#039;t make much sense, but in this example we are executing the OS command &amp;quot;uptime&amp;quot; on all cores&lt;br /&gt;
# of the nodes allocated to this job. The output from each core is written to the file info.log. We&lt;br /&gt;
# will find 56 lines of output in the file info.log, each created by the corresponding core executing&lt;br /&gt;
# the uptime command.&lt;br /&gt;
mpirun uptime &amp;gt; info.log&lt;br /&gt;
#&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
A good queue to test scripts is the &#039;&#039;&#039;&amp;quot;xeon28&amp;quot;&#039;&#039;&#039; queue. In the queue, we have 2 14-core Xeon processers per node, so that means that each node has 56 actual cores. We do not consider hyperthreading when doing parallel computing. 56 actual cores is what&#039;s being used here. The job submission will look like this:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
qsub -q xeon28 -l walltime=1:0:0 -l select=2:ncpus=28:mpiprocs=28:mem=60G parallel.job&lt;br /&gt;
        ^                  ^               ^       ^           ^      ^ ^ ^&lt;br /&gt;
        |                  |               |       |           |      | | + --- the name of the job script to execute&lt;br /&gt;
        |                  |               |       |           |      | + ----- don&#039;t forget to specify gigabytes&lt;br /&gt;
        |                  |               |       |           |      + ------- the amount of memory to request per node&lt;br /&gt;
        |                  |               |       |           + -------------- the number of MPI tasks per nodes&lt;br /&gt;
        |                  |               |       + -------------------------- the number of cores per node&lt;br /&gt;
        |                  |               + ---------------------------------- the number of nodes to select in the queue&lt;br /&gt;
        |                   + ------------------------------------------------- the requested time, in this case 1h&lt;br /&gt;
        + --------------------------------------------------------------------- the queue to be used for the job&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
At this point, the job has created a file &amp;quot;info.log&amp;quot; with 56 lines, one per core:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
 22:26:05 up 23 days,  9:28,  0 users,  load average: 0.00, 0.00, 0.00&lt;br /&gt;
 22:26:05 up 23 days,  9:28,  0 users,  load average: 0.00, 0.00, 0.00&lt;br /&gt;
 22:26:05 up 23 days,  9:28,  0 users,  load average: 0.00, 0.00, 0.00&lt;br /&gt;
 22:26:05 up 23 days,  9:28,  0 users,  load average: 0.00, 0.00, 0.00&lt;br /&gt;
 22:26:05 up 23 days,  9:28,  0 users,  load average: 0.00, 0.00, 0.00&lt;br /&gt;
 22:26:05 up 23 days,  9:28,  0 users,  load average: 0.00, 0.00, 0.00&lt;br /&gt;
 22:26:05 up 23 days,  9:53,  0 users,  load average: 0.06, 0.03, 0.00&lt;br /&gt;
 22:26:05 up 23 days,  9:53,  0 users,  load average: 0.06, 0.03, 0.00&lt;br /&gt;
 22:26:05 up 23 days,  9:53,  0 users,  load average: 0.06, 0.03, 0.00&lt;br /&gt;
...&lt;br /&gt;
&amp;lt;/PRE&amp;gt; &lt;br /&gt;
&lt;br /&gt;
In this simple example, the lines look all the same. Upon close examination through, you can find slightly different values for some of the lines. Some lines say that the machine is up for 23 days and 9:28, while others say 23 days and 9:53. Because all 28 cores of a node would see the same uptime of the server, half of the entries show one time stamp, and the other 28 cores show the other one. That demonstrates that the 56 processes have been running independently on 2 nodes.&lt;br /&gt;
&lt;br /&gt;
===Embedded Job Resource Requests===&lt;br /&gt;
&lt;br /&gt;
The job script can be modified to embed the resource requests in form of a series of &#039;&#039;&#039;#PBS&#039;&#039;&#039; statements at the beginning of the script file. This is a very common practice use at many HPC installations and job submission engines. Let&#039;s go back to the previous example where we run the script on two nodes in parallel. That is the &#039;&#039;&#039;&amp;quot;parallel.job&amp;quot;&#039;&#039;&#039; script file again:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
#!/bin/bash&lt;br /&gt;
#&lt;br /&gt;
#PBS -q xeon28&lt;br /&gt;
#PBS -l walltime=1:0:0&lt;br /&gt;
#PBS -l select=2:ncpus=28:mpiprocs=28:mem=60G&lt;br /&gt;
#&lt;br /&gt;
# the following ensures that you will change into the directory where you are&lt;br /&gt;
# submitting the job from.&lt;br /&gt;
cd $PBS_O_WORKDIR&lt;br /&gt;
#&lt;br /&gt;
# to execute a simple command on all of the cores of all of the nodes allocated to the job,&lt;br /&gt;
# we need to make one of the MPI versions available. Let&#039;s use one of the most up-to-date&lt;br /&gt;
# MPI library available on the cluster&lt;br /&gt;
module load intel/2024.2.0/mpi/2021.13&lt;br /&gt;
#&lt;br /&gt;
# now we are apply a few settings that ensure that the MPI library will use the highest-performing&lt;br /&gt;
# Infiniband Interconnect, as well as a few options to tell MPI how to interface nodes with&lt;br /&gt;
# each other and which specific Infiniband adapter to use. This is complex and requires in-depth&lt;br /&gt;
# knowledge of the QLogic Infiniband adapters we are using. It is unlikely that you will ever have to&lt;br /&gt;
# deal with these options, because the &amp;quot;module load&amp;quot; command for the engineering applications we provide&lt;br /&gt;
# on ARROW will handle all those details transparently without the user needing to understand the details.&lt;br /&gt;
export I_MPI_HYDRA_BOOTSTRAP=ssh&lt;br /&gt;
export MPI_DEVICE=rdma:ofa-v2-ib0&lt;br /&gt;
export UCX_NET_DEVICES=qib0:1&lt;br /&gt;
#&lt;br /&gt;
# it doesn&#039;t make much sense, but in this example we are executing the OS command &amp;quot;uptime&amp;quot; on all cores&lt;br /&gt;
# of the nodes allocated to this job. The output from each core is written to the file info.log. We&lt;br /&gt;
# will find 56 lines of output in the file info.log, each created by the corresponding core executing&lt;br /&gt;
# the uptime command.&lt;br /&gt;
mpirun uptime &amp;gt; info.log&lt;br /&gt;
#&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
If the resource requests are embedded within the file, they don&#039;t have to be specified on the command line any longer (the command line overrides the embedded specifications though). This may be convenient, because all the user has to do for job submission is the following:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
qsub parallel.job&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
Here is an example with more resource specifications and job settings that affect the behavior of the job:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
#!/bin/bash&lt;br /&gt;
#&lt;br /&gt;
#PBS -q xeon28&lt;br /&gt;
#PBS -l walltime=1:0:0&lt;br /&gt;
#PBS -l select=2:ncpus=28:mpiprocs=28:mem=60G&lt;br /&gt;
#PBS -A Account&lt;br /&gt;
#PBS -j oe&lt;br /&gt;
#PBS -N JobName&lt;br /&gt;
#PBS -e log.error&lt;br /&gt;
#PBS -o log.output&lt;br /&gt;
#PBS -m bae&lt;br /&gt;
#&lt;br /&gt;
...&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
I leave this to you as an exercise to figure out what the various options mean and how to specify them. There are many more, all documented in the PBS PRO manual (see above). Most of them are not terribly relevant and can be omitted.&lt;br /&gt;
&lt;br /&gt;
===Interactive Jobs===&lt;br /&gt;
&lt;br /&gt;
On ARROW, we don&#039;t restrict queues to be used only in batch mode. While &#039;&#039;&#039;batch mode&#039;&#039;&#039; is efficient for lining up a lot of work to be executed one after the other, ARROW has been designed to &#039;&#039;&#039;allow efficient model and software development in interactive mode&#039;&#039;&#039;. We have always ensured to have more computers than minimally needed to make it possible to dedicate resources to developers as needed, even if that means wasted CPU cycles. At times, we may ask you to limit the number of interactive jobs so that a large batch workload can be processed efficiently. This happens from time to time, and we have our users coordinate this with each other.&lt;br /&gt;
&lt;br /&gt;
Let&#039;s assume that you are developing an MPI application, or you are working on a complex &#039;&#039;&#039;OpenFOAM&#039;&#039;&#039; model that requires to start parallel processes over and over again just to find a bug and then fix it quickly. To do that, you can &#039;&#039;&#039;request an interactive job&#039;&#039;&#039; by adding the &#039;&#039;&#039;&amp;quot;-I&amp;quot;&#039;&#039;&#039; option to the job submission command (this is an uppercase I). Let&#039;s go to the parallel multi-node example from above:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
qsub -I -q xeon28 -l walltime=1:0:0 -l select=2:ncpus=28:mpiprocs=28:mem=60G&lt;br /&gt;
      ^    ^                  ^               ^       ^           ^      ^ ^&lt;br /&gt;
      |    |                  |               |       |           |      | + --- don&#039;t forget to specify gigabytes&lt;br /&gt;
      |    |                  |               |       |           |      + ----- the amount of memory to request per node&lt;br /&gt;
      |    |                  |               |       |           + ------------ the number of MPI tasks per nodes&lt;br /&gt;
      |    |                  |               |       + ------------------------ the number of cores per node&lt;br /&gt;
      |    |                  |               + -------------------------------- the number of nodes to select in the queue&lt;br /&gt;
      |    |                   + ----------------------------------------------- the requested time, in this case 1h&lt;br /&gt;
      |    + ------------------------------------------------------------------- the queue to be used for the job&lt;br /&gt;
      + ------------------------------------------------------------------------ request an interactive job &amp;lt;&amp;lt;===&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
When running interactive jobs with the &#039;&#039;&#039;&amp;quot;-I&amp;quot;&#039;&#039;&#039; parameter, we don&#039;t specify av job script at the end of the submission command. The interactive job will instead start (once the nodes are available) in interactive mode, meaning that the terminal session changes over from being a series of commands executed on the login server to being a series of commands being executed on the first node of the group of nodes that are allocated to the job. At this point, you can change to the desired working directory, but what you do with the allocated resources is entirely up to you. You can load modules, including MPI libraries, and then issue the commands for your application interactively and see how they execute. If you start an &#039;&#039;&#039;&amp;quot;mpirun&amp;quot;&#039;&#039;&#039;, the cores on your allocated secondary node will work as expected. There is no difference to batch mode, other than you having the ability to execute lines of commands at will.&lt;br /&gt;
&lt;br /&gt;
===Interactive Jobs with X-Windows GUI Applications===&lt;br /&gt;
&lt;br /&gt;
Interactive use can go further than that. With some of our software applications, like &#039;&#039;&#039;StarCCM+&#039;&#039;&#039;, you can run an &#039;&#039;&#039;interactive GUI application&#039;&#039;&#039; where you control the computational work from within the applications&#039; GUI. Within the GUI, you can control execution of the numerical solver that runs on as many cores as you requested, while being able to reconfigure the case through the GUI as well. Furthermore, you can visualize developing results on the fly by creating complex plots and visualizations.&lt;br /&gt;
&lt;br /&gt;
All that is need is an option &#039;&#039;&#039;&amp;quot;-X&amp;quot;&#039;&#039;&#039; being used as part of the job submission, like this:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
qsub -X -I -q xeon28 -l walltime=1:0:0 -l select=2:ncpus=28:mpiprocs=28:mem=60G&lt;br /&gt;
      ^  ^    ^                  ^               ^       ^           ^      ^ ^&lt;br /&gt;
      |  |    |                  |               |       |           |      | + --- don&#039;t forget to specify gigabytes&lt;br /&gt;
      |  |    |                  |               |       |           |      + ----- the amount of memory to request per node&lt;br /&gt;
      |  |    |                  |               |       |           + ------------ the number of MPI tasks per nodes&lt;br /&gt;
      |  |    |                  |               |       + ------------------------ the number of cores per node&lt;br /&gt;
      |  |    |                  |               + -------------------------------- the number of nodes to select in the queue&lt;br /&gt;
      |  |    |                   + ----------------------------------------------- the requested time, in this case 1h&lt;br /&gt;
      |  |    + ------------------------------------------------------------------- the queue to be used for the job&lt;br /&gt;
      |  + ------------------------------------------------------------------------ request an interactive job&lt;br /&gt;
      + --------------------------------------------------------------------------- request GUI capabilities &amp;lt;&amp;lt;===&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
===Running Multiple Jobs on Single Nodes===&lt;br /&gt;
&lt;br /&gt;
A feature that is new on ARROW is the ability to run multiple jobs on a single node. Let&#039;s assume that you are performing a sensitivity analysis on an existing model, and the model is simple enough to return results within a reasonable time on just a few cores of a higher end machine (maybe you are running SMP versions of &#039;&#039;&#039;LS-Dyna&#039;&#039;&#039;). Our high end machines have 64 cores, so lets assume you have an &#039;&#039;&#039;LS-Dyna&#039;&#039;&#039; model that runs well on 8 cores and doesn&#039;t use a whole lot of memory. In this case, you can submit individual jobs that request simply 8 cores and a fraction of the available memory available on the node, and all jobs execute independently from each other. Each job is fit into a slot where available. It is not very different from using whole nodes for everything. The important consideration is that each job is cleanly constrained into it&#039;s allotted resources using the &#039;&#039;&#039;CGROUPS&#039;&#039;&#039; functionality of modern operating systems. Because an abusive user cannot use more cores or more memory than allocated to his job, other users can safely run smaller jobs on the same node.&lt;br /&gt;
&lt;br /&gt;
Lets assume that we have a number of smaller jobs that we want to run on a single node in the &#039;&#039;&#039;&amp;quot;xeon28&amp;quot;&#039;&#039;&#039; queue. Each job would be submitted by using reduced resources that allow for sharing but that  guarantee that the jobs will be run successfully. In this case, you can &#039;&#039;&#039;submit many jobs&#039;&#039;&#039; in the following manner (with a job script for the small jobs, each of which can &#039;&#039;&#039;request varying resources&#039;&#039;&#039; if needed - some may want to run on 5 cores, others on 3):&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
#!/bin/bash&lt;br /&gt;
#&lt;br /&gt;
# the following ensures that you will change into the directory where you are&lt;br /&gt;
# submitting the job from.&lt;br /&gt;
cd $PBS_O_WORKDIR&lt;br /&gt;
#&lt;br /&gt;
# now we sleep for 300 seconds and waste time. This is a placeholder for your application,&lt;br /&gt;
# which would be doing useful work for you.&lt;br /&gt;
sleep 300&lt;br /&gt;
#&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
Now we submit a variety of these jobs (11 total in this example) to the &#039;&#039;&#039;&amp;quot;xeon28&amp;quot;&#039;&#039;&#039; queue for execution (note that the first few jobs request different amounts of memory and core counts):&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
qsub -q xeon28 -l walltime=1:0:0 -l select=1:ncpus=12:mpiprocs=12:mem=5G small.job &lt;br /&gt;
qsub -q xeon28 -l walltime=1:0:0 -l select=1:ncpus=10:mpiprocs=10:mem=7G small.job &lt;br /&gt;
qsub -q xeon28 -l walltime=1:0:0 -l select=1:ncpus=8:mpiprocs=8:mem=9G small.job &lt;br /&gt;
qsub -q xeon28 -l walltime=1:0:0 -l select=1:ncpus=16:mpiprocs=16:mem=20G small.job &lt;br /&gt;
qsub -q xeon28 -l walltime=1:0:0 -l select=1:ncpus=2:mpiprocs=2:mem=2G small.job &lt;br /&gt;
qsub -q xeon28 -l walltime=1:0:0 -l select=1:ncpus=2:mpiprocs=2:mem=2G small.job &lt;br /&gt;
qsub -q xeon28 -l walltime=1:0:0 -l select=1:ncpus=2:mpiprocs=2:mem=2G small.job &lt;br /&gt;
qsub -q xeon28 -l walltime=1:0:0 -l select=1:ncpus=2:mpiprocs=2:mem=2G small.job &lt;br /&gt;
qsub -q xeon28 -l walltime=1:0:0 -l select=1:ncpus=2:mpiprocs=2:mem=2G small.job &lt;br /&gt;
qsub -q xeon28 -l walltime=1:0:0 -l select=1:ncpus=2:mpiprocs=2:mem=2G small.job &lt;br /&gt;
qsub -q xeon28 -l walltime=1:0:0 -l select=1:ncpus=2:mpiprocs=2:mem=2G small.job &lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
They are now running in the order of submission, allocated on as few nodes in the &amp;quot;xeon28&amp;quot; queue as necessary. Only 2 nodes are being loaded quite heavily, and 4 more cores are in use on a third node.&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
Nov 20 23:34 ley@login3:myjobfolder$ qstat -an1&lt;br /&gt;
&lt;br /&gt;
pbs: &lt;br /&gt;
                                                            Req&#039;d  Req&#039;d   Elap&lt;br /&gt;
Job ID          Username Queue    Jobname    SessID NDS TSK Memory Time  S Time&lt;br /&gt;
--------------- -------- -------- ---------- ------ --- --- ------ ----- - -----&lt;br /&gt;
&lt;br /&gt;
3082.pbs        ley      xeon28   small.job  813221   1  12    5gb 01:00 R 00:01 p001/0*12&lt;br /&gt;
3083.pbs        ley      xeon28   small.job  813288   1  10    7gb 01:00 R 00:01 p001/1*10&lt;br /&gt;
3084.pbs        ley      xeon28   small.job  671792   1   8    9gb 01:00 R 00:01 p002/0*8&lt;br /&gt;
3085.pbs        ley      xeon28   small.job  671845   1  16   20gb 01:00 R 00:01 p002/1*16&lt;br /&gt;
3086.pbs        ley      xeon28   small.job  813361   1   2    2gb 01:00 R 00:00 p001/2*2&lt;br /&gt;
3087.pbs        ley      xeon28   small.job  813413   1   2    2gb 01:00 R 00:00 p001/3*2&lt;br /&gt;
3088.pbs        ley      xeon28   small.job  813464   1   2    2gb 01:00 R 00:00 p001/4*2&lt;br /&gt;
3089.pbs        ley      xeon28   small.job  671912   1   2    2gb 01:00 R 00:00 p002/2*2&lt;br /&gt;
3090.pbs        ley      xeon28   small.job  671969   1   2    2gb 01:00 R 00:00 p002/3*2&lt;br /&gt;
3091.pbs        ley      xeon28   small.job  632092   1   2    2gb 01:00 R 00:00 p003/0*2&lt;br /&gt;
3092.pbs        ley      xeon28   small.job  632100   1   2    2gb 01:00 R 00:00 p003/1*2&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
This is a particularly effective strategy to run concurrently many cases that don&#039;t scale well beyond a few cores. When running them on fewer cores but many of them at the same time, the overall processing rate will be much higher than executing them the traditional way.&lt;br /&gt;
&lt;br /&gt;
===Running Jobs using GPUs===&lt;br /&gt;
&lt;br /&gt;
The principle of running multiple jobs on a single node becomes particularly important when using servers equipped with &#039;&#039;&#039;GPUs&#039;&#039;&#039; for &#039;&#039;&#039;ML/AI&#039;&#039;&#039; applications. The cluster doesn&#039;t have a whole lot of &#039;&#039;&#039;GPUs&#039;&#039;&#039; at this point. We have three machines with three &#039;&#039;&#039;A4000&#039;&#039;&#039; GOUs, a &#039;&#039;&#039;total of 9 A4000 GPUs&#039;&#039;&#039;. Then we have a much more powerful single machine with our &#039;&#039;&#039;four A6000 GPUs&#039;&#039;&#039;.&lt;br /&gt;
&lt;br /&gt;
Using multiple GPUs in a single application is still something where the software has to be designed with hardware in mind. GPUs have several methods of communicating with each other, e.g. very fast &#039;&#039;&#039;NVLINK&#039;&#039;&#039; between pairs of GPUs, GPUs being directly connected to one of the two CPUs in the system and thus being able to communicate faster, and GPUs that have to jump between processors when communicating, and then the whole issue of having to go possibly through PCIe bridges.&lt;br /&gt;
&lt;br /&gt;
On our system, we are providing the ability to &#039;&#039;&#039;work mostly with individual GPUs&#039;&#039;&#039;. Users can also reserve entire nodes and develop or run applications that are adapted to that hardware, including several GPUs installed on that node. One thing we do not provide is the ability of GPU to GPU communication between nodes. Thus, a job cannot request more than one GPU node at a time.&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
qsub -q a4000 -I -l walltime=1:0:0 -l select=1:ncpus=5:mem=150G:ngpus=1&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
With these specifications, three single GPU jobs can run on a single server. Each job sees only one of the reserved GPU.&lt;br /&gt;
&lt;br /&gt;
To run a massive GPU job on 64 cores with 4 &#039;&#039;&#039;A6000 GPUs&#039;&#039;&#039;, submit the job like this:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
qsub -q a6000 -I -l walltime=1:0:0 -l select=1:ncpus=64:mem=725G:ngpus=4&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
===Manual pages for qsub===&lt;br /&gt;
&lt;br /&gt;
To learn more about the &amp;quot;qsub&amp;quot; command, you can use the command &amp;quot;man qsub&amp;quot;, which will print a lot of detailed information about the capabilities of this command.&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
$ man qsub&lt;br /&gt;
&lt;br /&gt;
qsub(1B)                                               PBS Professional                                              qsub(1B)&lt;br /&gt;
&lt;br /&gt;
NAME&lt;br /&gt;
       qsub - submit a job to PBS&lt;br /&gt;
&lt;br /&gt;
SYNOPSIS&lt;br /&gt;
       qsub [-a &amp;lt;date and time&amp;gt;] [-A &amp;lt;account string&amp;gt;] [-c &amp;lt;checkpoint spec&amp;gt;]&lt;br /&gt;
            [-C &amp;lt;directive prefix&amp;gt;] [-e &amp;lt;path&amp;gt;] [-f] [-h]&lt;br /&gt;
            [-I [-G [-- &amp;lt;GUI application/script&amp;gt;]] | [-X]] [-j &amp;lt;join&amp;gt;]&lt;br /&gt;
            [-J &amp;lt;range&amp;gt; [%&amp;lt;max subjobs]] [-k &amp;lt;discard&amp;gt;] [-l &amp;lt;resource list&amp;gt;]&lt;br /&gt;
            [-m &amp;lt;mail events&amp;gt;] [-M &amp;lt;user list&amp;gt;] [-N &amp;lt;name&amp;gt;] [-o &amp;lt;path&amp;gt;]&lt;br /&gt;
            [-p &amp;lt;priority&amp;gt;] [-P &amp;lt;project&amp;gt;] [-q &amp;lt;destination&amp;gt;] [-r &amp;lt;y|n&amp;gt;]&lt;br /&gt;
            [-R &amp;lt;remove options&amp;gt;] [-S &amp;lt;path list&amp;gt;] [-u &amp;lt;user list&amp;gt;]&lt;br /&gt;
            [-v &amp;lt;variable list&amp;gt;] [-V] [-W &amp;lt;additional attributes&amp;gt;] [-z]&lt;br /&gt;
            [- | &amp;lt;script&amp;gt; | -- &amp;lt;executable&amp;gt; [&amp;lt;arguments to executable&amp;gt;]]&lt;br /&gt;
       qsub --version&lt;br /&gt;
&lt;br /&gt;
DESCRIPTION&lt;br /&gt;
       You use the qsub command to submit a batch job to PBS.  Submitting a PBS job specifies a task, requests resources, and&lt;br /&gt;
       sets job attributes.&lt;br /&gt;
... &amp;lt;many more pages&amp;gt;&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
==LS-Dyna on the ARROW Cluster==&lt;br /&gt;
&lt;br /&gt;
===Currently Available LS-Dyna Versions===&lt;br /&gt;
&lt;br /&gt;
The following is a list of &#039;&#039;&#039;LS-Dyna versions&#039;&#039;&#039; available on &#039;&#039;&#039;ARROW&#039;&#039;&#039; after the latest reconfiguration of the system. As per LSTC/ANSYS, &#039;&#039;&#039;versions before 14.0.0 are not necessarily fully supported any longer&#039;&#039;&#039; because they are supposedly not compatible with modern operating systems and cannot be made to work reliably. We have tested the listed older versions of LS-Dyna and they have passed basic tests. They may not behave exactly as they did on the old CentOS 7 operating system, and time will show whether they can still be used or whether you will need to update your models and use a fully supported version.&lt;br /&gt;
&lt;br /&gt;
All versions are loaded using the &#039;&#039;&#039;&amp;quot;module load&amp;quot;&#039;&#039;&#039; command. Versions can be listed with the &#039;&#039;&#039;&amp;quot;module avail ls-dyna&amp;quot;&#039;&#039;&#039; command. To load one of the modules, use the following syntax:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
module load ls-dyna/14.2.0/mpi-d8-ifort190-avx512&lt;br /&gt;
            ^       ^      ^   ^  ^        ^&lt;br /&gt;
            |       |      |   |  |        + --- specify the extended instruction set needed for execution&lt;br /&gt;
            |       |      |   |  + ------------ load the version of the compiler that was used to create this &lt;br /&gt;
            |       |      |   + --------------- load the version that supports double precision variables&lt;br /&gt;
            |       |      + ------------------- load the MPP (MPI) version of LS-Dyna&lt;br /&gt;
            |       + -------------------------- load specifically version 14.2.0&lt;br /&gt;
            + ---------------------------------- load a version of LS-Dyna&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
The version string is composed of multiple elements to indicate variants in compilers and compiler options. Use the following guideline to choose an appropriate version to load:&lt;br /&gt;
&lt;br /&gt;
* &#039;&#039;&#039;&amp;quot;1&amp;quot;&#039;&#039;&#039; or &#039;&#039;&#039;&amp;quot;mpi&amp;quot;&#039;&#039;&#039; indicates whether this is a single node version of LS-Dyna (&#039;&#039;&#039;SMP&#039;&#039;&#039;) or whether this is a multi-node MPI version (&#039;&#039;&#039;MPP&#039;&#039;&#039;). All MPI versions use the &#039;&#039;&#039;IntelMPI 2022&#039;&#039;&#039; libraries which have been tested thoroughly on ARROW. MPI versions will use the Infiniband Network of ARROW for high-speed and low-latency inter-process communication using RDMA (remote direct memory access).&lt;br /&gt;
* All LS-Dyna versions are available in either &#039;&#039;&#039;floating point&#039;&#039;&#039; or &#039;&#039;&#039;double precision variants&#039;&#039;&#039;. Floating point variants use 4 bytes to represent a value, and double precision variants use 8 bytes. There are pros and cons for choosing one over the other variant. With regards to computational efficiency, both perform nearly the same because all machines are equipped with 64-bit CPUs.&lt;br /&gt;
** &#039;&#039;&#039;&amp;quot;f4&amp;quot;&#039;&#039;&#039; floating point versions&lt;br /&gt;
*** Pros: These require significantly less memory to run. Results occupy less disk space, and can be transferred significantly faster into and out of ARROW.&lt;br /&gt;
*** Cons: The numerical resolution is limited to 7 significant digits, which is often undesirable when dealing with mathematical operations on small and large numbers at the same time.&lt;br /&gt;
** &#039;&#039;&#039;&amp;quot;r8&amp;quot;&#039;&#039;&#039; double precision versions&lt;br /&gt;
*** Pros: The numerical resolution is about twice the number of significant digits compare to &amp;quot;f4&amp;quot;, which helps when when dealing with mathematical operations on small and large numbers at the same time.&lt;br /&gt;
*** Cons: These require more memory to run. Results occupy more disk space, and it takes longer to transfer data into and out of ARROW.&lt;br /&gt;
* There are two more identifiers to choose from when it comes to the variants of the executables: the specific compiler used to create the executable and the specific processor instruction set required for running the executable.&lt;br /&gt;
** For modern versions of LS-Dyna, two compilers have been used by the developers to create LS-Dyna executables: the &#039;&#039;&#039;Intel Fortran Compiler&#039;&#039;&#039; and the &#039;&#039;&#039;AOCC (AMD Optimizing C/C++ and Fortran) compiler&#039;&#039;&#039;. Both variants of the software are supported on ARROW. This gives users the opportunity to choose an alternate variant of the same LS-Dyna version when running into bugs or crashes.&lt;br /&gt;
** The variants based on the various instruction set extensions (&#039;&#039;&#039;SSE2&#039;&#039;&#039;, &#039;&#039;&#039;AVX2&#039;&#039;&#039;, &#039;&#039;&#039;AVX512&#039;&#039;&#039;, and so on) gives users even more options when choosing an alternate LS-Dyna variant of the same version when running into bugs or crashes. These instruction sets are mostly related to performance gains on specific processors. We have not performed thorough performance tests and cannot recommend specific versions right now.&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
$ module avail ls-dyna&lt;br /&gt;
--------------------------------------------- /shared/apps/modulefiles ---------------------------------------------&lt;br /&gt;
ls-dyna/09.3.1/1-d8-ifort131           ls-dyna/12.2.1/mpi-f4-ifort160-sse2    ls-dyna/14.2.0/mpi-f4-ifort190-avx512  &lt;br /&gt;
ls-dyna/09.3.1/1-f4-ifort131           ls-dyna/12.2.2/1-d8-ifort160-sse2      ls-dyna/14.2.0/mpi-f4-ifort190-sse2    &lt;br /&gt;
ls-dyna/09.3.1/mpi-d8-ifort131-avx2    ls-dyna/12.2.2/1-f4-ifort160-sse2      ls-dyna/15.0.2/1-d8-ifort190-sse2      &lt;br /&gt;
ls-dyna/09.3.1/mpi-d8-ifort131-avx512  ls-dyna/12.2.2/mpi-d8-aocc400-avx2     ls-dyna/15.0.2/1-f4-ifort190-sse2      &lt;br /&gt;
ls-dyna/09.3.1/mpi-f4-ifort131-avx2    ls-dyna/12.2.2/mpi-d8-ifort160-avx2    ls-dyna/15.0.2/mpi-d8-aocc400-avx2     &lt;br /&gt;
ls-dyna/09.3.1/mpi-f4-ifort131-avx512  ls-dyna/12.2.2/mpi-d8-ifort160-sse2    ls-dyna/15.0.2/mpi-d8-ifort190-avx2    &lt;br /&gt;
ls-dyna/10.2.0/1-d8-ifort160           ls-dyna/12.2.2/mpi-f4-aocc400-avx2     ls-dyna/15.0.2/mpi-d8-ifort190-avx512  &lt;br /&gt;
ls-dyna/10.2.0/1-f4-ifort160           ls-dyna/12.2.2/mpi-f4-ifort160-avx2    ls-dyna/15.0.2/mpi-d8-ifort190-sse2    &lt;br /&gt;
ls-dyna/11.0.0/1-d8-ifort160           ls-dyna/12.2.2/mpi-f4-ifort160-sse2    ls-dyna/15.0.2/mpi-f4-aocc400-avx2     &lt;br /&gt;
ls-dyna/11.0.0/1-f4-ifort160           ls-dyna/13.0.0/1-d8-ifort190           ls-dyna/15.0.2/mpi-f4-ifort190-avx2    &lt;br /&gt;
ls-dyna/11.1.0/1-d8-ifort160-sse2      ls-dyna/13.0.0/1-f4-ifort190           ls-dyna/15.0.2/mpi-f4-ifort190-avx512  &lt;br /&gt;
ls-dyna/11.1.0/1-f4-ifort160-sse2      ls-dyna/13.0.0/mpi-d8-ifort190-avx2    ls-dyna/15.0.2/mpi-f4-ifort190-sse2    &lt;br /&gt;
ls-dyna/11.2.0/1-d8-ifort160           ls-dyna/13.0.0/mpi-d8-ifort190-sse2    ls-dyna/16.0.0/1-d8-aocc420-avx2       &lt;br /&gt;
ls-dyna/11.2.0/1-f4-ifort160           ls-dyna/13.0.0/mpi-f4-ifort190-avx2    ls-dyna/16.0.0/1-d8-aocc420-avx512     &lt;br /&gt;
ls-dyna/11.2.0/mpi-f4-ifort160-avx2    ls-dyna/13.0.0/mpi-f4-ifort190-sse2    ls-dyna/16.0.0/1-d8-ifort190-sse2      &lt;br /&gt;
ls-dyna/11.2.0/mpi-f4-ifort160-sse2    ls-dyna/13.1.0/mpi-d8-aocc310-avx2     ls-dyna/16.0.0/1-f4-aocc420-avx2       &lt;br /&gt;
ls-dyna/11.2.1/1-d8-ifort160           ls-dyna/13.1.0/mpi-d8-ifort190-avx2    ls-dyna/16.0.0/1-f4-aocc420-avx512     &lt;br /&gt;
ls-dyna/11.2.1/1-f4-ifort160           ls-dyna/13.1.0/mpi-d8-ifort190-sse2    ls-dyna/16.0.0/1-f4-ifort190-sse2      &lt;br /&gt;
ls-dyna/11.2.1/mpi-d8-ifort160-avx2    ls-dyna/13.1.0/mpi-f4-aocc310-avx2     ls-dyna/16.0.0/mpi-d8-aocc420-avx2     &lt;br /&gt;
ls-dyna/11.2.1/mpi-d8-ifort160-sse2    ls-dyna/13.1.0/mpi-f4-ifort190-avx2    ls-dyna/16.0.0/mpi-d8-aocc420-avx512   &lt;br /&gt;
ls-dyna/11.2.1/mpi-f4-ifort160-avx2    ls-dyna/13.1.0/mpi-f4-ifort190-sse2    ls-dyna/16.0.0/mpi-d8-ifort190-avx2    &lt;br /&gt;
ls-dyna/11.2.1/mpi-f4-ifort160-sse2    ls-dyna/13.1.1/mpi-d8-ifort190-avx2    ls-dyna/16.0.0/mpi-d8-ifort190-avx512  &lt;br /&gt;
ls-dyna/11.2.2/mpi-d8-ifort160-avx2    ls-dyna/13.1.1/mpi-d8-ifort190-sse2    ls-dyna/16.0.0/mpi-d8-ifort190-sse2    &lt;br /&gt;
ls-dyna/11.2.2/mpi-d8-ifort160-sse2    ls-dyna/13.1.1/mpi-f4-ifort190-avx2    ls-dyna/16.0.0/mpi-f4-aocc420-avx2     &lt;br /&gt;
ls-dyna/11.2.2/mpi-f4-ifort160-avx2    ls-dyna/13.1.1/mpi-f4-ifort190-sse2    ls-dyna/16.0.0/mpi-f4-aocc420-avx512   &lt;br /&gt;
ls-dyna/11.2.2/mpi-f4-ifort160-sse2    ls-dyna/14.0.0/1-d8-ifort190           ls-dyna/16.0.0/mpi-f4-ifort190-avx2    &lt;br /&gt;
ls-dyna/12.1.0/1-d8-ifort160           ls-dyna/14.0.0/1-f4-ifort190           ls-dyna/16.0.0/mpi-f4-ifort190-avx512  &lt;br /&gt;
ls-dyna/12.1.0/1-f4-aocc310            ls-dyna/14.0.0/mpi-d8-aocc310-avx2     ls-dyna/16.0.0/mpi-f4-ifort190-sse2    &lt;br /&gt;
ls-dyna/12.1.0/1-f4-ifort160           ls-dyna/14.0.0/mpi-d8-ifort190-avx2    ls-dyna/16.1.0/mpi-d8-aocc420-avx2     &lt;br /&gt;
ls-dyna/12.1.0/mpi-d8-aocc310-avx2     ls-dyna/14.0.0/mpi-d8-ifort190-sse2    ls-dyna/16.1.0/mpi-d8-aocc420-avx512   &lt;br /&gt;
ls-dyna/12.1.0/mpi-d8-ifort160-avx2    ls-dyna/14.0.0/mpi-f4-ifort190-avx2    ls-dyna/16.1.0/mpi-d8-ifort190-avx2    &lt;br /&gt;
ls-dyna/12.1.0/mpi-d8-ifort160-sse2    ls-dyna/14.0.0/mpi-f4-ifort190-sse2    ls-dyna/16.1.0/mpi-d8-ifort190-avx512  &lt;br /&gt;
ls-dyna/12.1.0/mpi-f4-aocc310-avx2     ls-dyna/14.1.0/1-d8-ifort190-sse2      ls-dyna/16.1.0/mpi-d8-ifort190-sse2    &lt;br /&gt;
ls-dyna/12.1.0/mpi-f4-ifort160-avx2    ls-dyna/14.1.0/1-f4-ifort190-sse2      ls-dyna/16.1.0/mpi-f4-aocc420-avx2     &lt;br /&gt;
ls-dyna/12.1.0/mpi-f4-ifort160-sse2    ls-dyna/14.1.0/mpi-d8-aocc400-avx2     ls-dyna/16.1.0/mpi-f4-aocc420-avx512&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
===Submitting an LS-Dyna Job===&lt;br /&gt;
&lt;br /&gt;
&amp;lt;BLOCKQUOTE&amp;gt;&lt;br /&gt;
[[File:Attention.jpg|25px]] &#039;&#039;&#039;IMPORTANT NOTE:&#039;&#039;&#039; The job/queue manager can now track the number of LS-Dyna licenses given out to individual&lt;br /&gt;
jobs. At submission time, it is not possible to know what software a user may run. But by adding the clause &amp;quot;-l dynalic&amp;quot; at submission time,&lt;br /&gt;
the queue manager can calculate the total number of cores required and keep track of LS-Dyna licenses used by the job. When loading a version of LS-Dyna, a check will be performed, and LS-Dyna will be prevented from running if the &amp;quot;-l dynalic&amp;quot; clause was not used when submitting the job.&#039;&#039;&lt;br /&gt;
&lt;br /&gt;
&amp;lt;!--&lt;br /&gt;
&#039;&#039;The job/queue manager can track the number of LS-Dyna licenses to some degree. If all &#039;&#039;&#039;LS-Dyna users&#039;&#039;&#039; cooperate and use a script like the one shown below when submitting their jobs, the total number of concurrent &#039;&#039;&#039;LS-Dyna licenses&#039;&#039;&#039; will be tracked by the job manager correctly. That means that users can submit any number of LS-Dyna jobs, and jobs will only start when a sufficient number of licenses is available. This is managed by the &#039;&#039;&#039;&amp;quot;dynalic&amp;quot;&#039;&#039;&#039; resource at the end of the select statement. In this example, a 2-node job on 64-core nodes will need a total of &#039;&#039;&#039;&amp;quot;dynalic=128&amp;quot;&#039;&#039;&#039; licenses. This accounting breaks down when users don&#039;t use the &#039;&#039;&#039;&amp;quot;dynalic=XXX&amp;quot;&#039;&#039;&#039; statement, or when they don&#039;t calculate the number of licenses correctly. In that case, LS-Dyna jobs of all users are subject to sudden failure when LS-Dyna licenses run out. Please understand the importance of this specific setting in your job.&#039;&#039;&lt;br /&gt;
--&amp;gt;&lt;br /&gt;
&amp;lt;/BLOCKQUOTE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
Furthermore, careful consideration should be given with regards to choice of resources for an &#039;&#039;&#039;LS-Dyna job&#039;&#039;&#039;. With 64 cores available on a single node in the &#039;&#039;&#039;&amp;quot;epyc1&amp;quot;&#039;&#039;&#039; and &#039;&#039;&#039;&amp;quot;epyc2&amp;quot;&#039;&#039;&#039; queues, it may be counterproductive to run a job on two nodes instead of a single node. Users should run their jobs with different numbers of nodes and determine whether performance increases. It may well decrease when running a job on two or more nodes. The outcome of such tests will tell what the best allocation of resources will be.&lt;br /&gt;
&lt;br /&gt;
Most users use a job script like the following. All methods for job submission the the previous chapters apply as well, so there is a lot of flexibility:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
#!/bin/bash&lt;br /&gt;
#&lt;br /&gt;
#PBS -q epyc1&lt;br /&gt;
#PBS -l walltime=12:0:0&lt;br /&gt;
#PBS -l select=2:ncpus=64:mpiprocs=64:mem=225G,dynalic&lt;br /&gt;
#PBS -N JobName&lt;br /&gt;
#PBS -e log.error&lt;br /&gt;
#PBS -o log.output&lt;br /&gt;
#&lt;br /&gt;
cd $PBS_O_WORKDIR&lt;br /&gt;
#&lt;br /&gt;
module load ls-dyna/12.2.1/mpi-f4-ifort160-avx2&lt;br /&gt;
module load dynamore/current&lt;br /&gt;
#&lt;br /&gt;
mpirun ls-dyna i=main.k memory1=300m memory2=100m &amp;gt; dyna.log&lt;br /&gt;
#&lt;br /&gt;
# when using the Dynamore tools, you can start something like this at the end&lt;br /&gt;
DM.plotcprs.lnx -merge &amp;gt;&amp;gt; dyna.log&lt;br /&gt;
#&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
===LSTC Tools: LS-OPT and LS-PREPOST===&lt;br /&gt;
&lt;br /&gt;
For the new Rocky 9 cluster, I have not looked deeply into the ls-opt and ls-prepost packages that were installed. I noticed though that the LSTC server provided access to much newer versions of both software packages. If you would like to learn more or have a specific version in mind, I can easily download and install it for you.&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
$ module avail ls-opt&lt;br /&gt;
----------------------------------------------- /shared/apps/modulefiles ------------------------------------------------&lt;br /&gt;
ls-opt/5.1.1  ls-opt/6.0.0  ls-opt/7.0.0  ls-opt/7.0.2   ls-opt/2022R2  &lt;br /&gt;
ls-opt/5.2.1  ls-opt/6.1.0  ls-opt/7.0.1  ls-opt/2022R1  ls-opt/2023R1  &lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
To start the software, type:&lt;br /&gt;
 lsopt&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
$ module avail ls-prepost&lt;br /&gt;
----------------------------------------------- /shared/apps/modulefiles ------------------------------------------------&lt;br /&gt;
ls-prepost/4.5.10  ls-prepost/4.8.13  ls-prepost/4.8.30  ls-prepost/4.9.16  ls-prepost/4.10.7 &lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
To start the software, type:&lt;br /&gt;
 lsprepost&lt;br /&gt;
&lt;br /&gt;
===Dynamore Software===&lt;br /&gt;
&lt;br /&gt;
The Dynamore tools are available as a module:&lt;br /&gt;
&lt;br /&gt;
 module load dynamore/current&lt;br /&gt;
&lt;br /&gt;
We typically acquire a yearly license for the tools as we purchase licenses for LS-Dyna.&lt;br /&gt;
&lt;br /&gt;
===Vendor License File Installation===&lt;br /&gt;
&lt;br /&gt;
If you would like for us to install a vendor license for LS-Dyna models, please contact us for the required information. We can send you the general LS-Dyna license file to show the host ids for the license server. Using that information, your vendor should be able to create a vendor license file. Please send that file to us per Email or by other means.&lt;br /&gt;
&lt;br /&gt;
==StarCCM+ on the ARROW Cluster==&lt;br /&gt;
&lt;br /&gt;
===Currently Available StarCCM+ Versions===&lt;br /&gt;
&lt;br /&gt;
As of late 2025, we have the following versions of &#039;&#039;&#039;StarCCM+&#039;&#039;&#039; available on the cluster:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
$ module avail starccm&lt;br /&gt;
---------------------------- /shared/apps/modulefiles ----------------------------&lt;br /&gt;
starccm/15.02.007-R8  starccm/16.06.008-R8  starccm/18.06.006-R8  &lt;br /&gt;
starccm/15.02.009-R8  starccm/17.02.007-R8  starccm/19.02.009-R8  &lt;br /&gt;
starccm/15.04.008-R8  starccm/17.02.008-R8  starccm/20.04.007-R8  &lt;br /&gt;
starccm/15.06.008-R8  starccm/17.04.007-R8  starccm/20.06.007-R8  &lt;br /&gt;
starccm/16.02.008-R8  starccm/17.06.007-R8  &lt;br /&gt;
starccm/16.04.007-R8  starccm/18.04.008-R8&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
If using a &#039;&#039;&#039;single node&#039;&#039;&#039; for StarCCM+, job submission (for an interactive job) is simple and will use appropriate default settings:&lt;br /&gt;
&lt;br /&gt;
 qsub -I -X -q epyc1 -l walltime=20:00:00&lt;br /&gt;
&lt;br /&gt;
StarCCM+ can make use of the job scheduler attributes by automatically obtaining the number of cores and other resources from OpenPBS. In this case, the default number of cores and mpi processes for StarCCM+ are both 64 when using the epyc1 queue. So you can start your StarCCM+ run with:&lt;br /&gt;
&lt;br /&gt;
 module load starccm/15.02.007-R8 (or any other version)&lt;br /&gt;
 starccm+ -bs pbs&lt;br /&gt;
&lt;br /&gt;
In this case, there is no need for StarCCM+ to be told to run the case in parallel with the selected number of cores/mpiprocs.&lt;br /&gt;
&lt;br /&gt;
This can get a bit more complex when running on multiple nodes or when requesting high memory nodes. In that case you would use job submission parameters as shown below:&lt;br /&gt;
&lt;br /&gt;
 qsub -I -X -q epyc1 -l walltime=20:00:00,select=2:ncpus=64:mpiprocs=64:mem=500GB&lt;br /&gt;
&lt;br /&gt;
Requesting nodes that can satisfy those resources, two nodes with these attributes must exist. We have multiple nodes with 512GB in the epyc1 queue, meaning that this job will run on two machines that have at least the required amount of memory installed (on each node). The job will be queued until two machines like this will be available. If no machines with these resources exist, the job will stay in the queue forever. Therefore, you have to craft the submission string carefully.&lt;br /&gt;
&lt;br /&gt;
To accommodate high memory jobs, the nodes have been assigned priorities for assignment. Low memory jobs have the highest priority and will be assigned to nodes that can accommodate the request. High memory nodes have the lowest priority, meaning that they are the last ones given out to users. This makes it more likely that a high memory job can be run soon when the cluster is moderately loaded with jobs.&lt;br /&gt;
&lt;br /&gt;
StarCCM+ will always use the Intel MPI fabric. Other MPI versions do not work, even when selected on the command line.&lt;br /&gt;
&lt;br /&gt;
==OpenFOAM on the ARROW Cluster==&lt;br /&gt;
&lt;br /&gt;
===Currently Available OpenFOAM  Versions===&lt;br /&gt;
&lt;br /&gt;
As of late 2025, we have the following versions of OpenFOAM available on the cluster:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
$ module avail openfoam&lt;br /&gt;
------------ /shared/apps/modulefiles ------------&lt;br /&gt;
openfoam/9   openfoam/13      openfoam/v2312  &lt;br /&gt;
openfoam/10  openfoam/13-amd  openfoam/v2406  &lt;br /&gt;
openfoam/11  openfoam/v2212   &lt;br /&gt;
openfoam/12  openfoam/v2306&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
Contact us if you encounter problems; there can be various reasons why OpenFOAM may have trouble on certain hardware or when compiling dynamic code. When loading OpenFOAM modules, a number of dependencies will be automatically loaded for you, and you don&#039;t have to load those yourself. For example:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
$ module load openfoam/13&lt;br /&gt;
Loading openfoam/13&lt;br /&gt;
  Loading requirement: intel/2024.2.0/mpi/2021.13 gcc/gcc-12.1.0&lt;br /&gt;
&lt;br /&gt;
$ module list&lt;br /&gt;
Currently Loaded Modulefiles:&lt;br /&gt;
 1) intel/2024.2.0/mpi/2021.13   2) gcc/gcc-12.1.0   3) openfoam/13&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
In this case, OpenFOAM 13 loads the Intel 2024 MPI module, and loads the GCC compiler 12.1. OpenFOAM was compiled from source, and has been compiled specifically with that compiler and MPI version, so it make little sense to use other compilers or MPI libraries.&lt;br /&gt;
&lt;br /&gt;
Note: We have found a problem with running the Intel 2024 MPI library in the amd64 queue. Therefore, we have a modified module that uses the Intel 2022 library (I know -- Intel 2022 gives you the 2021 MPI libraries, but that is the way Intel distributes this software):&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
$ module load openfoam/13-amd &lt;br /&gt;
Loading mpi version 2021.7.0&lt;br /&gt;
Loading openfoam/13-amd&lt;br /&gt;
  Loading requirement: intel/2022.2.0/mpi/2021.7.0 gcc/gcc-12.1.0&lt;br /&gt;
&lt;br /&gt;
$ module list&lt;br /&gt;
Currently Loaded Modulefiles:&lt;br /&gt;
 1) intel/2022.2.0/mpi/2021.7.0   2) gcc/gcc-12.1.0   3) openfoam/13-amd&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
If you are compiling OpenFOAM yourself, the modules are of little help. You would need to select the appropriate MPI version and compiler before doing so, and then consistently load them before running your OpenFOAM executables. Within the &amp;quot;etc/bashrc&amp;quot; file in the source code tree, you want to set the MPI library to INTELMPI. As usual with self-compiled versions of OpenFOAM, you would &amp;quot;source etc/bashrc&amp;quot; to set up your personal environment to run your home-brew version of OpenFOAM. Contact us if you need to learn more about compiling OpenFOAM on the system.&lt;br /&gt;
&lt;br /&gt;
==Additional Software Applications and Libraries==&lt;br /&gt;
&lt;br /&gt;
===Loadable GCC Compiler Versions===&lt;br /&gt;
&lt;br /&gt;
The Rocky 9.6 operating system uses the GCC 11.5 compiler. That should be sufficient for most users when compiling your own applications. In case you need to use either a more up-to-date compiler, or if you need an older compiler for compatibility, we make the following versions available as loadable modules.&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
$ module avail gcc&lt;br /&gt;
------------ /shared/apps/modulefiles ------------&lt;br /&gt;
gcc/gcc-4.9.4  gcc/gcc-7.5.0  gcc/gcc-10.3.0  &lt;br /&gt;
gcc/gcc-5.5.0  gcc/gcc-8.5.0  gcc/gcc-11.3.0  &lt;br /&gt;
gcc/gcc-6.5.0  gcc/gcc-9.5.0  gcc/gcc-12.1.0&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
Additional versions can be created and made available as modules as well. If you need a specific version that is not currently available, please ask us to compiler and install it. If necessary, we may be able to provide access to other compilers, for example LLVM. We do not provide access to proprietary compilers at this time.&lt;br /&gt;
&lt;br /&gt;
===MPI Libraries and Runtimes===&lt;br /&gt;
&lt;br /&gt;
While we seem to have a variety of MPI versions and flavors available to users, the only MPI versions that allow us to run software over Infiniband are the Intel MPI libraries. Some of the installed alternatives are likely to fail, or will have a set of environment variables that have to be set. All major engineering software packages that we offer are pre-configured with specific MPI versions and settings that have been tested and/or provided by the vendors.&lt;br /&gt;
&lt;br /&gt;
Note: Some MPI libraries may seem to work. They may allow your MPI application to run. But inter-process network communication may travel through the rather slow and high-latency Ethernet fabric, making MPI applications very ineffective and are probably not worth while.&lt;br /&gt;
&lt;br /&gt;
===MatLab Runtimes===&lt;br /&gt;
&lt;br /&gt;
We can install MatLAB run time libraries as needed and have them available as loadable modules. Recently, we had a problem with MatLAB run time libraries being identified as security vulnerabilities. Contact us if you need them installed for one of your projects.&lt;br /&gt;
&lt;br /&gt;
===Anaconda and variants (miniconda etc)===&lt;br /&gt;
&lt;br /&gt;
Our current practice is to have users download and install their own versions of Anaconda and its variants in their own home directories. This allows for maximum flexibility when it comes to installable software modules, and users can maintain the installation, upgrades, and maintenance themselves. If you encounter issues, please contact us. One known side effect of Anaconda installations is a performance hit when starting your software, e.g. python scripts may take 30 seconds or more to execute. This is an artefact caused by the Lustre file system, which has been designed for large files accessible from many machines simultaneously. Performance on reading many small files has not been considered and is fairly poor. Again, contact us and we will design a solution for you as needed.&lt;/div&gt;</summary>
		<author><name>Ley</name></author>
	</entry>
	<entry>
		<id>https://wiki.anl.gov/wiki_tracc/index.php?title=Job_Submission_and_Monitoring&amp;diff=2492</id>
		<title>Job Submission and Monitoring</title>
		<link rel="alternate" type="text/html" href="https://wiki.anl.gov/wiki_tracc/index.php?title=Job_Submission_and_Monitoring&amp;diff=2492"/>
		<updated>2026-02-26T18:15:38Z</updated>

		<summary type="html">&lt;p&gt;Ley: /* The Queue Summary Script (qsum) */&lt;/p&gt;
&lt;hr /&gt;
&lt;div&gt;{| align=&amp;quot;right&amp;quot;&lt;br /&gt;
| __TOC__&lt;br /&gt;
|}&lt;br /&gt;
==Resource Summary View==&lt;br /&gt;
&lt;br /&gt;
To get started, users can query the overall status of resources on the cluster. The &#039;&#039;&#039;&amp;quot;qsum&amp;quot;&#039;&#039;&#039; script will list all queues and nodes, as well as how many are offline, down, free, or assigned to users. This is a script developed by our team, and may need to be updated if something goes wrong. Please contact us if you experience any problems.&lt;br /&gt;
&lt;br /&gt;
Each queue groups a number of nodes together based on their hardware and software configurations. Nodes can be part of more than one queue, and there are other complex details that we are ignoring here for the purpose of keeping it simple.&lt;br /&gt;
&lt;br /&gt;
===Queues===&lt;br /&gt;
&lt;br /&gt;
Here is a very brief summary of what each of the queues is, and how to use them efficiently:&lt;br /&gt;
&lt;br /&gt;
; a4000: This is a queue that has three machines with 16 cores each; each of these machines is furthermore equipped with three A4000 GPUs. That makes a total of 9 A4000 GPUs available to users. Neither the GPUs nor the processors are particularly powerful these days, but they make for a good software development environment. The machines have 512GB of memory, which makes them a good platform for experimenting with GPU capabilities.&lt;br /&gt;
; a6000: This is a queue that has only one single machine with 64 cores total, and is equipped with four A6000 GPUs. The system can be upgraded to 8 A6000 GPUs if needed. This is a decent GPU machine that can take a solid workload. The machine has 750GB of memory, which makes for a good production platform.&lt;br /&gt;
; amd16: This is a queue with many of our older AMD-based 16-core machines, each of which equipped with 32GB of memory. While individual machines are a bit outdated, they are all interconnected with Infiniband and can provide a solid production workload in multi-node jobs over MPI without blocking the more current (and thus expensive) systems.&lt;br /&gt;
; epyc1/epyc2: These are 2 separate queues with slightly different performance characteristics. Each of the groups is interconnected with Infiniband to provide a platform for large and demanding software packages, such as LS-Dyna and StarCCM+. They have between 256GB and 512GB of memory. Because licenses for these software packages (LS-Dyna and StarCCM+) are very expensive, these applications should use the two epyc queues for making optimum use of limited core licenses available to each package.&lt;br /&gt;
; xeon28: This is a set of intermediate machines with 28 cores and 64GB of memory. They can be used for a variety of purposes, including MPI jobs and single node application software.&lt;br /&gt;
; virtual: This is a set of nodes without MPI capabilities. They are virtual machines with 32GB each. They can be used for higher demand interactive applications that would interfere otherwise with other users on the login node machines. A user would submit interactive jobs to individual virtual machines and thus avoid any significant load on login nodes.&lt;br /&gt;
&lt;br /&gt;
===The Queue Summary Script (qsum)===&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
$ qsum&lt;br /&gt;
=============== a4000 ==========================================================&lt;br /&gt;
Queue: &amp;quot;a4000&amp;quot; / nodes: 3 / down: 0 / offline: 0 / busy: 0 / available: 3&lt;br /&gt;
      AVAILABLE (3): g001, g002, g003&lt;br /&gt;
=============== a6000 ==========================================================&lt;br /&gt;
Queue: &amp;quot;a6000&amp;quot; / nodes: 1 / down: 0 / offline: 0 / busy: 0 / available: 1&lt;br /&gt;
      AVAILABLE (1): lambda01&lt;br /&gt;
=============== amd16 ==========================================================&lt;br /&gt;
Queue: &amp;quot;amd16&amp;quot; / nodes: 98 / down: 28 / offline: 5 / busy: 4 / available: 61&lt;br /&gt;
          DOWN (28): n012, n017, n018, n022, n024, n030, n034, n037, n038, n040&lt;br /&gt;
                     n041, n042, n056, n057, n060, n062, n064, n069, n071, n072&lt;br /&gt;
                     n076, n079, n080, n086, n087, n089, n093, n094&lt;br /&gt;
        OFFLINE (5): n001, n002, n003, n004, n010&lt;br /&gt;
       ac.xliu1 (4): n026, n027, n028, n029&lt;br /&gt;
     AVAILABLE (61): n005, n006, n007, n008, n009, n011, n013, n014, n015, n016&lt;br /&gt;
                     n019, n020, n021, n023, n025, n031, n032, n033, n035, n036&lt;br /&gt;
                     n039, n043, n044, n045, n046, n047, n048, n049, n050, n051&lt;br /&gt;
                     n052, n053, n054, n055, n058, n059, n061, n063, n065, n066&lt;br /&gt;
                     n067, n068, n070, n073, n074, n075, n077, n078, n081, n082&lt;br /&gt;
                     n083, n084, n085, n088, n090, n091, n095, n096, n097, n098&lt;br /&gt;
                     n099&lt;br /&gt;
=============== epyc1 ==========================================================&lt;br /&gt;
Queue: &amp;quot;epyc1&amp;quot; / nodes: 27 / down: 1 / offline: 0 / busy: 10 / available: 16&lt;br /&gt;
           DOWN (1): a011&lt;br /&gt;
       ac.ge.w* (2): a004, a027&lt;br /&gt;
       ac.razm* (2): a015, a016&lt;br /&gt;
       ac.vpra* (4): a005, a006, a008, a009&lt;br /&gt;
         msitek (2): a001, a002&lt;br /&gt;
     AVAILABLE (16): a003, a007, a010, a012, a013, a014, a017, a018, a019, a020&lt;br /&gt;
                     a021, a022, a023, a024, a025, a026&lt;br /&gt;
=============== epyc2 ==========================================================&lt;br /&gt;
Queue: &amp;quot;epyc2&amp;quot; / nodes: 20 / down: 0 / offline: 0 / busy: 16 / available: 4&lt;br /&gt;
      cbojano* (16): a028, a030, a031, a032, a033, a034, a035, a036, a037, a038&lt;br /&gt;
                     a039, a040, a044, a045, a046, a047&lt;br /&gt;
      AVAILABLE (4): a029, a041, a042, a043&lt;br /&gt;
=============== virtual ========================================================&lt;br /&gt;
Queue: &amp;quot;virtual&amp;quot; / nodes: 6 / down: 0 / offline: 0 / busy: 0 / available: 6&lt;br /&gt;
      AVAILABLE (6): v001, v002, v003, v004, v005, v006&lt;br /&gt;
=============== xeon28 =========================================================&lt;br /&gt;
Queue: &amp;quot;xeon28&amp;quot; / nodes: 12 / down: 0 / offline: 0 / busy: 0 / available: 12&lt;br /&gt;
     AVAILABLE (12): p001, p002, p003, p004, p005, p006, p007, p008, p009, p010&lt;br /&gt;
                     p011, p012&lt;br /&gt;
================================================================================&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
==Queue Status and Monitoring Jobs==&lt;br /&gt;
&lt;br /&gt;
===qstat===&lt;br /&gt;
&lt;br /&gt;
To find out out about the status of all running jobs on the cluster you can use the &#039;&#039;&#039;&amp;quot;qstat&amp;quot;&#039;&#039;&#039; command. Here is an example:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
$ qstat&lt;br /&gt;
&lt;br /&gt;
Nov 20 18:30 ley@login3:Plots$ qstat&lt;br /&gt;
Job id            Name             User              Time Use S Queue&lt;br /&gt;
----------------  ---------------- ----------------  -------- - -----&lt;br /&gt;
3023.pbs          STDIN            msitek            4144:14* R epyc2           &lt;br /&gt;
3029.pbs          STDIN            ley               76:46:53 R epyc2           &lt;br /&gt;
3032.pbs          STDIN            msitek            2879:52* R epyc2           &lt;br /&gt;
3033.pbs          STDIN            msitek            3687:29* R epyc2           &lt;br /&gt;
3048.pbs          foo.sh           james.cook               0 Q amd16           &lt;br /&gt;
3060.pbs          of13.sh          ley               310:47:* R epyc2           &lt;br /&gt;
3061.pbs          of13.sh          ley               308:37:* R epyc2           &lt;br /&gt;
3062.pbs          of13.sh          ley               308:02:* R epyc2           &lt;br /&gt;
3063.pbs          of13.sh          ley               308:15:* R epyc2&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
The first column shows the &#039;&#039;&#039;job id&#039;&#039;&#039;, a unique identifier for all jobs ever submitted to the cluster. This job id is important when killing jobs, or for other actions you may need to take.&lt;br /&gt;
&lt;br /&gt;
The next column shows the name of the job script. If the column shows &#039;&#039;&#039;STDIN&#039;&#039;&#039;, it means that this is an &#039;&#039;&#039;interactive job&#039;&#039;&#039; where a user can enter commands in a terminal window. This is particularly useful for model and software development task where the application has to be started and killed repeatedly.&lt;br /&gt;
&lt;br /&gt;
The owner of the job is shown next. These are the user names of the various people using the cluster.&lt;br /&gt;
&lt;br /&gt;
The last three columns indicate the current run time of the job, whether it is running (R) or waiting (Q) for execution. The last entry shows the queue in which the job is running.&lt;br /&gt;
&lt;br /&gt;
===qstat -an1===&lt;br /&gt;
&lt;br /&gt;
Adding a few options gives much more detail about each jobs.&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
qstat -an1&lt;br /&gt;
&lt;br /&gt;
Nov 20 13:09 ley@login3:Plots$ qstat -an1&lt;br /&gt;
&lt;br /&gt;
pbs: &lt;br /&gt;
                                                            Req&#039;d  Req&#039;d   Elap&lt;br /&gt;
Job ID          Username Queue    Jobname    SessID NDS TSK Memory Time  S Time&lt;br /&gt;
--------------- -------- -------- ---------- ------ --- --- ------ ----- - -----&lt;br /&gt;
3023.pbs        msitek   epyc2    STDIN      24360*   1  64  350gb 100:0 R 81:46 a028/0*64&lt;br /&gt;
3029.pbs        ley      epyc2    STDIN      21719*   2 128  100gb 200:0 R 72:31 a030/0*64+a031/0*64&lt;br /&gt;
3032.pbs        msitek   epyc2    STDIN      18102*   1  64  350gb 100:0 R 57:57 a029/0*64&lt;br /&gt;
3033.pbs        msitek   epyc2    STDIN      830486   1  64  350gb 100:0 R 57:53 a032/0*64&lt;br /&gt;
3048.pbs        james.c* amd16    foo.sh        --    1  28   30gb 01:00 Q   --   -- &lt;br /&gt;
3060.pbs        ley      epyc2    STDIN      763101   1  64  350gb 48:00 R 06:42 a033/0*64&lt;br /&gt;
3061.pbs        ley      epyc2    STDIN      763947   1  64  350gb 48:00 R 06:40 a034/0*64&lt;br /&gt;
3062.pbs        ley      epyc2    STDIN      761473   1  64  350gb 48:00 R 06:39 a035/0*64&lt;br /&gt;
3063.pbs        ley      epyc2    STDIN      766205   1  64  350gb 48:00 R 06:40 a036/0*64&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
In this table, you can see how many nodes and cores are being used by each job. For example, job 3029 of the user &amp;quot;ley&amp;quot; shows that it is running on 2 nodes using a total of 128 cores. In addition to the elapsed time, the table also show the reserved time for this job. This allow you to estimate when a job will be definitely finalized (or killed by the system if still running).&lt;br /&gt;
&lt;br /&gt;
The last column (without a header) is written because the option &#039;&#039;&#039;&amp;quot;-an1&amp;quot;&#039;&#039;&#039; was used. This useful to learn about which nodes are used by each job.&lt;br /&gt;
&lt;br /&gt;
===qstat -q===&lt;br /&gt;
&lt;br /&gt;
To learn more about the queues on the cluster, the &#039;&#039;&#039;&amp;quot;-q&amp;quot;&#039;&#039;&#039; option turns out to be useful.&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
$ qstat -q&lt;br /&gt;
&lt;br /&gt;
server: pbs&lt;br /&gt;
&lt;br /&gt;
Queue            Memory CPU Time Walltime Node   Run   Que   Lm  State&lt;br /&gt;
---------------- ------ -------- -------- ---- ----- ----- ----  -----&lt;br /&gt;
virtual            30gb    --       --       1     0     0   --   E R&lt;br /&gt;
a4000             500gb    --       --       1     0     0   --   E R&lt;br /&gt;
a6000             750gb    --       --       1     0     0   --   E R&lt;br /&gt;
xeon28             --      --       --       4     0     0   --   E R&lt;br /&gt;
amd16              --      --       --       8     0     1   --   E R&lt;br /&gt;
epyc2              --      --       --       2    14     0   --   E R&lt;br /&gt;
epyc1              --      --       --       2     0     0   --   E R&lt;br /&gt;
                                               ----- -----&lt;br /&gt;
                                                  14     1&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
For each queue, some basic values are displayed. The first three queues listed above have a default memory allocation as shown, and the column &#039;&#039;&#039;&amp;quot;Node&amp;quot;&#039;&#039;&#039; indicates the maximum number of nodes that can be asked for at job submission time. For example, you can request just one node for a job from the first three queues (because these are queues where MPI makes no sense). The &#039;&#039;&#039;&amp;quot;xeon28&amp;quot;&#039;&#039;&#039; queue also for a maximum of 4 nodes per MPI job. The &#039;&#039;&#039;&amp;quot;amd16&amp;quot;&#039;&#039;&#039; queue has a maximum of 8 nodes per job, and the &#039;&#039;&#039;&amp;quot;epyc1&amp;quot;&#039;&#039;&#039; and &#039;&#039;&#039;&amp;quot;epyc2&amp;quot;&#039;&#039;&#039; queues have maxima of two nodes per job. These limitations can be changed by the administrator as needed. As shown above, this will prevent inefficient resource requests.&lt;br /&gt;
&lt;br /&gt;
===qstat -f===&lt;br /&gt;
&lt;br /&gt;
The command &#039;&#039;&#039;&amp;quot;qstat -f -F json 3029&amp;quot;&#039;&#039;&#039; retrieves extremely detailed stats on the running job 3029. The result can be returned in JSON format to be ready for further processing (shown below).&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
$ qstat -f -F json 3029&lt;br /&gt;
{&lt;br /&gt;
    &amp;quot;timestamp&amp;quot;:1763705353,&lt;br /&gt;
    &amp;quot;pbs_version&amp;quot;:&amp;quot;23.06.06&amp;quot;,&lt;br /&gt;
    &amp;quot;pbs_server&amp;quot;:&amp;quot;pbs&amp;quot;,&lt;br /&gt;
    &amp;quot;Jobs&amp;quot;:{&lt;br /&gt;
        &amp;quot;3029.pbs&amp;quot;:{&lt;br /&gt;
            &amp;quot;Job_Name&amp;quot;:&amp;quot;STDIN&amp;quot;,&lt;br /&gt;
            &amp;quot;Job_Owner&amp;quot;:&amp;quot;ley@login4&amp;quot;,&lt;br /&gt;
            &amp;quot;resources_used&amp;quot;:{&lt;br /&gt;
                &amp;quot;cpupercent&amp;quot;:98,&lt;br /&gt;
                &amp;quot;cput&amp;quot;:&amp;quot;76:46:53&amp;quot;,&lt;br /&gt;
                &amp;quot;hpmem&amp;quot;:&amp;quot;0b&amp;quot;,&lt;br /&gt;
                &amp;quot;mem&amp;quot;:&amp;quot;52428800kb&amp;quot;,&lt;br /&gt;
                &amp;quot;ncpus&amp;quot;:128,&lt;br /&gt;
                &amp;quot;vmem&amp;quot;:&amp;quot;52428800kb&amp;quot;,&lt;br /&gt;
                &amp;quot;walltime&amp;quot;:&amp;quot;78:09:32&amp;quot;&lt;br /&gt;
            },&lt;br /&gt;
            &amp;quot;job_state&amp;quot;:&amp;quot;R&amp;quot;,&lt;br /&gt;
            &amp;quot;queue&amp;quot;:&amp;quot;epyc2&amp;quot;,&lt;br /&gt;
            &amp;quot;server&amp;quot;:&amp;quot;pbs&amp;quot;,&lt;br /&gt;
            &amp;quot;Checkpoint&amp;quot;:&amp;quot;u&amp;quot;,&lt;br /&gt;
            &amp;quot;ctime&amp;quot;:&amp;quot;Mon Nov 17 17:58:25 2025&amp;quot;,&lt;br /&gt;
            &amp;quot;Error_Path&amp;quot;:&amp;quot;/dev/pts/0&amp;quot;,&lt;br /&gt;
            &amp;quot;exec_host&amp;quot;:&amp;quot;a030/0*64+a031/0*64&amp;quot;,&lt;br /&gt;
            &amp;quot;exec_vnode&amp;quot;:&amp;quot;(a030:ncpus=64:mem=52428800kb)+(a031:ncpus=64:mem=52428800kb)&amp;quot;,&lt;br /&gt;
            &amp;quot;Hold_Types&amp;quot;:&amp;quot;n&amp;quot;,&lt;br /&gt;
            &amp;quot;interactive&amp;quot;:&amp;quot;True&amp;quot;,&lt;br /&gt;
            &amp;quot;Join_Path&amp;quot;:&amp;quot;n&amp;quot;,&lt;br /&gt;
            &amp;quot;Keep_Files&amp;quot;:&amp;quot;n&amp;quot;,&lt;br /&gt;
            &amp;quot;Mail_Points&amp;quot;:&amp;quot;a&amp;quot;,&lt;br /&gt;
            &amp;quot;mtime&amp;quot;:&amp;quot;Fri Nov 21 00:07:59 2025&amp;quot;,&lt;br /&gt;
            &amp;quot;Output_Path&amp;quot;:&amp;quot;/dev/pts/0&amp;quot;,&lt;br /&gt;
            &amp;quot;Priority&amp;quot;:0,&lt;br /&gt;
            &amp;quot;qtime&amp;quot;:&amp;quot;Mon Nov 17 17:58:25 2025&amp;quot;,&lt;br /&gt;
            &amp;quot;Rerunable&amp;quot;:&amp;quot;False&amp;quot;,&lt;br /&gt;
            &amp;quot;Resource_List&amp;quot;:{&lt;br /&gt;
                &amp;quot;mem&amp;quot;:&amp;quot;100gb&amp;quot;,&lt;br /&gt;
                &amp;quot;mpiprocs&amp;quot;:128,&lt;br /&gt;
                &amp;quot;ncpus&amp;quot;:128,&lt;br /&gt;
                &amp;quot;nodect&amp;quot;:2,&lt;br /&gt;
                &amp;quot;place&amp;quot;:&amp;quot;free&amp;quot;,&lt;br /&gt;
                &amp;quot;select&amp;quot;:&amp;quot;2:ncpus=64:mem=50gb:mpiprocs=64&amp;quot;,&lt;br /&gt;
                &amp;quot;walltime&amp;quot;:&amp;quot;200:00:00&amp;quot;&lt;br /&gt;
            },&lt;br /&gt;
            &amp;quot;stime&amp;quot;:&amp;quot;Mon Nov 17 17:58:25 2025&amp;quot;,&lt;br /&gt;
            &amp;quot;session_id&amp;quot;:2171964,&lt;br /&gt;
            &amp;quot;jobdir&amp;quot;:&amp;quot;/mnt/lustre/arrow/home/ley&amp;quot;,&lt;br /&gt;
            &amp;quot;substate&amp;quot;:42,&lt;br /&gt;
            &amp;quot;Variable_List&amp;quot;:{&lt;br /&gt;
                &amp;quot;PBS_O_HOME&amp;quot;:&amp;quot;/mnt/lustre/arrow/home/ley&amp;quot;,&lt;br /&gt;
                &amp;quot;PBS_O_LANG&amp;quot;:&amp;quot;en_US.UTF-8&amp;quot;,&lt;br /&gt;
                &amp;quot;PBS_O_LOGNAME&amp;quot;:&amp;quot;ley&amp;quot;,&lt;br /&gt;
                &amp;quot;PBS_O_PATH&amp;quot;:&amp;quot;/shared/apps/active/lstc/lsprepost/SP-4.5:/shared/apps/active/lstc/lsprepost/DP-4.3:/shared/bin:/usr/share/Modules/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/pbs/bin:/opt/thinlinc/bin:/opt/thinlinc/sbin:/mnt/lustre/arrow/home/ley/.local/bin:/mnt/lustre/arrow/home/ley/bin&amp;quot;,&lt;br /&gt;
                &amp;quot;PBS_O_MAIL&amp;quot;:&amp;quot;/var/spool/mail/ley&amp;quot;,&lt;br /&gt;
                &amp;quot;PBS_O_SHELL&amp;quot;:&amp;quot;/bin/bash&amp;quot;,&lt;br /&gt;
                &amp;quot;PBS_O_WORKDIR&amp;quot;:&amp;quot;/mnt/lustre/arrow/home/ley/Qualification/LS-Dyna/Rocky9/Seatbelt/Template&amp;quot;,&lt;br /&gt;
                &amp;quot;PBS_O_SYSTEM&amp;quot;:&amp;quot;Linux&amp;quot;,&lt;br /&gt;
                &amp;quot;PBS_O_QUEUE&amp;quot;:&amp;quot;epyc2&amp;quot;,&lt;br /&gt;
                &amp;quot;PBS_O_HOST&amp;quot;:&amp;quot;login4&amp;quot;&lt;br /&gt;
            },&lt;br /&gt;
            &amp;quot;comment&amp;quot;:&amp;quot;Job run at Mon Nov 17 at 17:58 on (a030:ncpus=64:mem=52428800kb)+(a031:ncpus=64:mem=52428800kb)&amp;quot;,&lt;br /&gt;
            &amp;quot;etime&amp;quot;:&amp;quot;Mon Nov 17 17:58:25 2025&amp;quot;,&lt;br /&gt;
            &amp;quot;run_count&amp;quot;:1,&lt;br /&gt;
            &amp;quot;Submit_arguments&amp;quot;:&amp;quot;-I -q epyc2 -l walltime=200:00:00,select=2:ncpus=64:mem=50gb:mpiprocs=64&amp;quot;,&lt;br /&gt;
            &amp;quot;project&amp;quot;:&amp;quot;_pbs_project_default&amp;quot;,&lt;br /&gt;
            &amp;quot;Submit_Host&amp;quot;:&amp;quot;login4&amp;quot;&lt;br /&gt;
        }&lt;br /&gt;
    }&lt;br /&gt;
}&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
===Manual pages for qstat===&lt;br /&gt;
&lt;br /&gt;
To learn more about the &#039;&#039;&#039;&amp;quot;qstat&amp;quot;&#039;&#039;&#039; command, you can use the command &#039;&#039;&#039;&amp;quot;man qstat&amp;quot;&#039;&#039;&#039;, which will print a lot of detailed information about the capabilities of this command.&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
$ man qstat&lt;br /&gt;
&lt;br /&gt;
qstat(1B)                                       PBS Professional                                       qstat(1B)&lt;br /&gt;
&lt;br /&gt;
NAME&lt;br /&gt;
       qstat - display status of PBS jobs, queues, or servers&lt;br /&gt;
&lt;br /&gt;
SYNOPSIS&lt;br /&gt;
       Displaying Job Status&lt;br /&gt;
              Default format:&lt;br /&gt;
              qstat [-E] [-J] [-p] [-t] [-w] [-x] [[&amp;lt;job ID&amp;gt; | &amp;lt;destination&amp;gt;] ...]&lt;br /&gt;
&lt;br /&gt;
              Long format:&lt;br /&gt;
              qstat -f [-F json | dsv [-D &amp;lt;delimiter&amp;gt;]] [-E] [-J] [-p] [-t] [-w]&lt;br /&gt;
                    [-x] [[&amp;lt;job ID&amp;gt; | &amp;lt;destination&amp;gt;] ...]&lt;br /&gt;
... &amp;lt;many more pages&amp;gt;&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
==Job Submission Basics==&lt;br /&gt;
&lt;br /&gt;
Jobs are submitted into the system using the &#039;&#039;&#039;&amp;quot;qsub&amp;quot;&#039;&#039;&#039; application. This application can take many different options and allows for a lot of different resource requests to tell the cluster what to do. We are running &#039;&#039;&#039;OpenPBS 23.06.06&#039;&#039;&#039; as our job scheduler. Here is a link to the User&#039;s Manual (of PBS PRO) if you want to explore gory details and capabilities. The User&#039;s Guide has about 240 pages, the Reference Guide has 500 pages, and the Big Book has 2500 pages. So there is a lot of information available. I also added job submission info for the LCRC cluster.&lt;br /&gt;
&lt;br /&gt;
* [https://argonne-lcrc.github.io/user-guides/running-jobs-at-lcrc/pbs-pro/ Argonne&#039;s LCRC pages on job submissions on their clusters]&lt;br /&gt;
* [https://help.altair.com/2022.1.0/PBS%20Professional/PBSUserGuide2022.1.pdf PBS Professional 2022.1 User&#039;s Guide]&lt;br /&gt;
* [https://help.altair.com/2022.1.0/PBS%20Professional/PBSReferenceGuide2022.1.pdf PBS Professional 2022.1 Reference Guide]&lt;br /&gt;
* [https://help.altair.com/2022.1.0/PBS%20Professional/PBS2022.1.pdf Altair PBS Professional 2022.1 Big Book]&lt;br /&gt;
&lt;br /&gt;
The User&#039;s Guide can be very helpful to clarify some of the concepts and capabilities, but it can be hard to find the specific information you may be looking for. Please understand that we are no longer running TORQUE and MAUI, so the syntax for job submission is distinctively different yet quite similar.&lt;br /&gt;
&lt;br /&gt;
The reference guide may be helpful to understand the complete syntax and full capabilities of the software.&lt;br /&gt;
&lt;br /&gt;
The big book is what I had to use when configuring OpenPBS earlier this year. This includes all the tricky details needed to make the system work smoothly for us. It&#039;s a bit scary to look at a PDF file that is 2500 pages long, but that is nothing compared to the StarCCM+ manuals.&lt;br /&gt;
&lt;br /&gt;
&amp;lt;BLOCKQUOTE&amp;gt;&lt;br /&gt;
[[File:Attention.jpg|25px]] &#039;&#039;&#039;IMPORTANT NOTE:&#039;&#039;&#039; &#039;&#039;The following sections are important to understand. They explain how jobs are submitted and then scjeduled for execution based on resources available and the specific need of the user.&#039;&#039;&lt;br /&gt;
&amp;lt;/BLOCKQUOTE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
The following sections explain the various tasks you may want to submit fir execution, ordered from simple to complex.&lt;br /&gt;
&lt;br /&gt;
* General Batch Jobs&lt;br /&gt;
** Requesting a Single Node for a Job&lt;br /&gt;
** Requesting Multiple Nodes for a Job&lt;br /&gt;
* Embedded Job Resource Requests&lt;br /&gt;
* Interactive Jobs&lt;br /&gt;
* Interactive Jobs with X-Windows GUI Applications&lt;br /&gt;
* Running Multiple Jobs on Single Nodes&lt;br /&gt;
* Running Jobs using GPUs&lt;br /&gt;
&lt;br /&gt;
===General Batch Jobs===&lt;br /&gt;
&lt;br /&gt;
Let&#039;s get started with a very basic usage of the system. Let&#039;s assume you have a simple application, and you want to execute it on a cluster node. Let&#039;s also assume that this is a very simple application, one that runs on one or a few cores, doesn&#039;t require any keyboard interaction with the user, doesn&#039;t need the user to see what&#039;s typically written to the screen, and writes its output to files. In this case, we can submit this application as a batch job, which will place it into an execution queue and process it as soon as a node becomes available.&lt;br /&gt;
&lt;br /&gt;
If the application requires more cores than a single node can provide, we can run the application over Infiniband with MPI message passing. In this case, we need to understand the concept of MPI applications a bit better. In both cases, we get started by creating a folder on the file system. Naming conventions are important so that you can distinguish the jobs by folder name.&lt;br /&gt;
&lt;br /&gt;
For both of the above scenarios, you would typically create a Bash shell script, and then submit the script into one of the queues for eventual execution.&lt;br /&gt;
&lt;br /&gt;
====Requesting a Single Node for a Job====&lt;br /&gt;
&lt;br /&gt;
Let&#039;s try something rather trivial to get used to the concept. Create yourself a folder, for example &#039;&#039;&#039;&amp;quot;myjobfolder&amp;quot;&#039;&#039;&#039;. Within that folder, create a job submission script. That script can have any name, but something short and simple may be best. Let&#039;s assume you create a file called &#039;&#039;&#039;&amp;quot;cluster.job&amp;quot;&#039;&#039;&#039;. The file doesn&#039;t have to have that extension. Any file name will do. But using the same filename for all of your jobs helps finding your way around the many files that will be created over time. The &#039;&#039;&#039;&amp;quot;cluster.job&amp;quot;&#039;&#039;&#039; file should look something like this:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
#!/bin/bash&lt;br /&gt;
#&lt;br /&gt;
# the following ensures that you will change into the directory where you are&lt;br /&gt;
# submitting the job from.&lt;br /&gt;
cd $PBS_O_WORKDIR&lt;br /&gt;
#&lt;br /&gt;
# now we sleep for 60 seconds and waste time. This is a placeholder for your application,&lt;br /&gt;
# which would be doing useful work for you.&lt;br /&gt;
sleep 60&lt;br /&gt;
#&lt;br /&gt;
# and after doing things, we may want to write something into a file to show that&lt;br /&gt;
# our jobs is done.&lt;br /&gt;
echo `date` &amp;gt; info.log&lt;br /&gt;
#&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
This can be submitted without detailed resource specifications (except for the walltime, which is be default 0:00:00):&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
$ qsub -q virtual -l walltime=1:00:00 cluster.job &lt;br /&gt;
3072.pbs&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
Wait a little, then check the status of running jobs:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
$ qstat -an1&lt;br /&gt;
&lt;br /&gt;
pbs: &lt;br /&gt;
                                                            Req&#039;d  Req&#039;d   Elap&lt;br /&gt;
Job ID          Username Queue    Jobname    SessID NDS TSK Memory Time  S Time&lt;br /&gt;
--------------- -------- -------- ---------- ------ --- --- ------ ----- - -----&lt;br /&gt;
3023.pbs        msitek   epyc2    STDIN      24360*   1  64  350gb 100:0 R 83:17 a028/0*64&lt;br /&gt;
3029.pbs        ley      epyc2    STDIN      21719*   2 128  100gb 200:0 R 74:00 a030/0*64+a031/0*64&lt;br /&gt;
3033.pbs        msitek   epyc2    STDIN      830486   1  64  350gb 100:0 R 59:23 a032/0*64&lt;br /&gt;
3048.pbs        james.c* amd16    foo.sh        --    1  28   30gb 01:00 Q   --   -- &lt;br /&gt;
3060.pbs        ley      epyc2    STDIN      763101   1  64  350gb 48:00 R 08:10 a033/0*64&lt;br /&gt;
3061.pbs        ley      epyc2    STDIN      763947   1  64  350gb 48:00 R 08:10 a034/0*64&lt;br /&gt;
3070.pbs        ley      epyc2    STDIN      766847   1  64  350gb 48:00 R 07:23 a042/0*64&lt;br /&gt;
3072.pbs        ley      virtual  cluster.j* 230230   1   4   30gb 01:00 E 00:01 v001/0*4&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
In this particular example, we are sending this job to the &#039;&#039;&#039;queue &amp;quot;virtual&amp;quot;&#039;&#039;&#039;. This queue, by default, allocates 30GB of memory to the job, and runs on 1 node with 4 cores. This is sufficient capacity to run quite a workload. When submitting a job to a single node, &#039;&#039;&#039;reasonable maximum allocations are automatically assigned&#039;&#039;&#039;, and the user doesn&#039;t have to worry about running out of memory or how many cores he will be using.&lt;br /&gt;
&lt;br /&gt;
The only required argument is the &#039;&#039;&#039;&amp;quot;walltime&amp;quot;&#039;&#039;&#039; argument. By default, the job will quit as soon as it is submitted. This indicates to the user that he forgot to provide the &#039;&#039;&#039;&amp;quot;walltime&amp;quot;&#039;&#039;&#039; argument.&lt;br /&gt;
&lt;br /&gt;
When the job disappears from the job list, it is done. At this point, you will find the file &amp;quot;info.log&amp;quot; in your job folder.&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
$ cat info.log &lt;br /&gt;
Thu Nov 20 08:00:31 PM CST 2025&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
====Requesting Multiple Nodes for a Job====&lt;br /&gt;
&lt;br /&gt;
To run jobs on multiple nodes, you will be likely &#039;&#039;&#039;executing jobs using MPI&#039;&#039;&#039;, the message passing interface. This establishes high-speed low-latency interconnections between the cores on one machine and the cores on the other machines. Data transfer does not require involvement of the cores themselves. Instead, the core tell the InfiniBand interconnect (and cores on the same node through shared memory) to transfer the data through RDMA, remoted direct memory access. The cores don&#039;t need to spend CPU cycles on copying data, but rather simply access the data once it has been copied by the Infiniband fabric. This makes for extremely efficient remote memory access, and message passing is used to coordinate data transfer between the cores no matter where they are located on any of the nodes.&lt;br /&gt;
&lt;br /&gt;
On our cluster, MPI-aware applications like &#039;&#039;&#039;OpenFOAM&#039;&#039;&#039;, &#039;&#039;&#039;StarCCM+&#039;&#039;&#039;, and &#039;&#039;&#039;LS-Dyna&#039;&#039;&#039; can be loaded as modules, which then automatically selects the most appropriate MPI library to use. The software applications have been tested to ensure that they work out-of-the box if a user selects any specific version of any of the applications.&lt;br /&gt;
&lt;br /&gt;
The following is a very trivial example for the MPI execution of a very simple executable, with one copy running on each core of the nodes allocated to  the job. It doesn&#039;t perform any real work and just wastes resources for a short time, but it illustrates how execution on the cores of various nodes works.&lt;br /&gt;
&lt;br /&gt;
Like in the previous section, we start with a simple job script that we submit to an appropriate queues. In this case, we pick a queue that has machines with Infiniband interfaces supporting efficient communications. Let&#039;s assume we edit a file with the name &#039;&#039;&#039;&amp;quot;parallel.job&amp;quot;&#039;&#039;&#039; like this:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
#!/bin/bash&lt;br /&gt;
#&lt;br /&gt;
# the following ensures that you will change into the directory where you are&lt;br /&gt;
# submitting the job from.&lt;br /&gt;
cd $PBS_O_WORKDIR&lt;br /&gt;
#&lt;br /&gt;
# to execute a simple command on all of the cores of all of the nodes allocated to the job,&lt;br /&gt;
# we need to make one of the MPI versions available. Let&#039;s use one of the most up-to-date&lt;br /&gt;
# MPI library available on the cluster&lt;br /&gt;
module load intel/2024.2.0/mpi/2021.13&lt;br /&gt;
#&lt;br /&gt;
# now we are apply a few settings that ensure that the MPI library will use the highest-performing&lt;br /&gt;
# Infiniband Interconnect, as well as a few options to tell MPI how to interface nodes with&lt;br /&gt;
# each other and which specific Infiniband adapter to use. This is complex and requires in-depth&lt;br /&gt;
# knowledge of the QLogic Infiniband adapters we are using. It is unlikely that you will ever have to&lt;br /&gt;
# deal with these options, because the &amp;quot;module load&amp;quot; command for the engineering applications we provide&lt;br /&gt;
# on ARROW will handle all those details transparently without the user needing to understand the details.&lt;br /&gt;
export I_MPI_HYDRA_BOOTSTRAP=ssh&lt;br /&gt;
export MPI_DEVICE=rdma:ofa-v2-ib0&lt;br /&gt;
export UCX_NET_DEVICES=qib0:1&lt;br /&gt;
#&lt;br /&gt;
# it doesn&#039;t make much sense, but in this example we are executing the OS command &amp;quot;uptime&amp;quot; on all cores&lt;br /&gt;
# of the nodes allocated to this job. The output from each core is written to the file info.log. We&lt;br /&gt;
# will find 56 lines of output in the file info.log, each created by the corresponding core executing&lt;br /&gt;
# the uptime command.&lt;br /&gt;
mpirun uptime &amp;gt; info.log&lt;br /&gt;
#&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
A good queue to test scripts is the &#039;&#039;&#039;&amp;quot;xeon28&amp;quot;&#039;&#039;&#039; queue. In the queue, we have 2 14-core Xeon processers per node, so that means that each node has 56 actual cores. We do not consider hyperthreading when doing parallel computing. 56 actual cores is what&#039;s being used here. The job submission will look like this:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
qsub -q xeon28 -l walltime=1:0:0 -l select=2:ncpus=28:mpiprocs=28:mem=60G parallel.job&lt;br /&gt;
        ^                  ^               ^       ^           ^      ^ ^ ^&lt;br /&gt;
        |                  |               |       |           |      | | + --- the name of the job script to execute&lt;br /&gt;
        |                  |               |       |           |      | + ----- don&#039;t forget to specify gigabytes&lt;br /&gt;
        |                  |               |       |           |      + ------- the amount of memory to request per node&lt;br /&gt;
        |                  |               |       |           + -------------- the number of MPI tasks per nodes&lt;br /&gt;
        |                  |               |       + -------------------------- the number of cores per node&lt;br /&gt;
        |                  |               + ---------------------------------- the number of nodes to select in the queue&lt;br /&gt;
        |                   + ------------------------------------------------- the requested time, in this case 1h&lt;br /&gt;
        + --------------------------------------------------------------------- the queue to be used for the job&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
At this point, the job has created a file &amp;quot;info.log&amp;quot; with 56 lines, one per core:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
 22:26:05 up 23 days,  9:28,  0 users,  load average: 0.00, 0.00, 0.00&lt;br /&gt;
 22:26:05 up 23 days,  9:28,  0 users,  load average: 0.00, 0.00, 0.00&lt;br /&gt;
 22:26:05 up 23 days,  9:28,  0 users,  load average: 0.00, 0.00, 0.00&lt;br /&gt;
 22:26:05 up 23 days,  9:28,  0 users,  load average: 0.00, 0.00, 0.00&lt;br /&gt;
 22:26:05 up 23 days,  9:28,  0 users,  load average: 0.00, 0.00, 0.00&lt;br /&gt;
 22:26:05 up 23 days,  9:28,  0 users,  load average: 0.00, 0.00, 0.00&lt;br /&gt;
 22:26:05 up 23 days,  9:53,  0 users,  load average: 0.06, 0.03, 0.00&lt;br /&gt;
 22:26:05 up 23 days,  9:53,  0 users,  load average: 0.06, 0.03, 0.00&lt;br /&gt;
 22:26:05 up 23 days,  9:53,  0 users,  load average: 0.06, 0.03, 0.00&lt;br /&gt;
...&lt;br /&gt;
&amp;lt;/PRE&amp;gt; &lt;br /&gt;
&lt;br /&gt;
In this simple example, the lines look all the same. Upon close examination through, you can find slightly different values for some of the lines. Some lines say that the machine is up for 23 days and 9:28, while others say 23 days and 9:53. Because all 28 cores of a node would see the same uptime of the server, half of the entries show one time stamp, and the other 28 cores show the other one. That demonstrates that the 56 processes have been running independently on 2 nodes.&lt;br /&gt;
&lt;br /&gt;
===Embedded Job Resource Requests===&lt;br /&gt;
&lt;br /&gt;
The job script can be modified to embed the resource requests in form of a series of &#039;&#039;&#039;#PBS&#039;&#039;&#039; statements at the beginning of the script file. This is a very common practice use at many HPC installations and job submission engines. Let&#039;s go back to the previous example where we run the script on two nodes in parallel. That is the &#039;&#039;&#039;&amp;quot;parallel.job&amp;quot;&#039;&#039;&#039; script file again:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
#!/bin/bash&lt;br /&gt;
#&lt;br /&gt;
#PBS -q xeon28&lt;br /&gt;
#PBS -l walltime=1:0:0&lt;br /&gt;
#PBS -l select=2:ncpus=28:mpiprocs=28:mem=60G&lt;br /&gt;
#&lt;br /&gt;
# the following ensures that you will change into the directory where you are&lt;br /&gt;
# submitting the job from.&lt;br /&gt;
cd $PBS_O_WORKDIR&lt;br /&gt;
#&lt;br /&gt;
# to execute a simple command on all of the cores of all of the nodes allocated to the job,&lt;br /&gt;
# we need to make one of the MPI versions available. Let&#039;s use one of the most up-to-date&lt;br /&gt;
# MPI library available on the cluster&lt;br /&gt;
module load intel/2024.2.0/mpi/2021.13&lt;br /&gt;
#&lt;br /&gt;
# now we are apply a few settings that ensure that the MPI library will use the highest-performing&lt;br /&gt;
# Infiniband Interconnect, as well as a few options to tell MPI how to interface nodes with&lt;br /&gt;
# each other and which specific Infiniband adapter to use. This is complex and requires in-depth&lt;br /&gt;
# knowledge of the QLogic Infiniband adapters we are using. It is unlikely that you will ever have to&lt;br /&gt;
# deal with these options, because the &amp;quot;module load&amp;quot; command for the engineering applications we provide&lt;br /&gt;
# on ARROW will handle all those details transparently without the user needing to understand the details.&lt;br /&gt;
export I_MPI_HYDRA_BOOTSTRAP=ssh&lt;br /&gt;
export MPI_DEVICE=rdma:ofa-v2-ib0&lt;br /&gt;
export UCX_NET_DEVICES=qib0:1&lt;br /&gt;
#&lt;br /&gt;
# it doesn&#039;t make much sense, but in this example we are executing the OS command &amp;quot;uptime&amp;quot; on all cores&lt;br /&gt;
# of the nodes allocated to this job. The output from each core is written to the file info.log. We&lt;br /&gt;
# will find 56 lines of output in the file info.log, each created by the corresponding core executing&lt;br /&gt;
# the uptime command.&lt;br /&gt;
mpirun uptime &amp;gt; info.log&lt;br /&gt;
#&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
If the resource requests are embedded within the file, they don&#039;t have to be specified on the command line any longer (the command line overrides the embedded specifications though). This may be convenient, because all the user has to do for job submission is the following:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
qsub parallel.job&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
Here is an example with more resource specifications and job settings that affect the behavior of the job:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
#!/bin/bash&lt;br /&gt;
#&lt;br /&gt;
#PBS -q xeon28&lt;br /&gt;
#PBS -l walltime=1:0:0&lt;br /&gt;
#PBS -l select=2:ncpus=28:mpiprocs=28:mem=60G&lt;br /&gt;
#PBS -A Account&lt;br /&gt;
#PBS -j oe&lt;br /&gt;
#PBS -N JobName&lt;br /&gt;
#PBS -e log.error&lt;br /&gt;
#PBS -o log.output&lt;br /&gt;
#PBS -m bae&lt;br /&gt;
#&lt;br /&gt;
...&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
I leave this to you as an exercise to figure out what the various options mean and how to specify them. There are many more, all documented in the PBS PRO manual (see above). Most of them are not terribly relevant and can be omitted.&lt;br /&gt;
&lt;br /&gt;
===Interactive Jobs===&lt;br /&gt;
&lt;br /&gt;
On ARROW, we don&#039;t restrict queues to be used only in batch mode. While &#039;&#039;&#039;batch mode&#039;&#039;&#039; is efficient for lining up a lot of work to be executed one after the other, ARROW has been designed to &#039;&#039;&#039;allow efficient model and software development in interactive mode&#039;&#039;&#039;. We have always ensured to have more computers than minimally needed to make it possible to dedicate resources to developers as needed, even if that means wasted CPU cycles. At times, we may ask you to limit the number of interactive jobs so that a large batch workload can be processed efficiently. This happens from time to time, and we have our users coordinate this with each other.&lt;br /&gt;
&lt;br /&gt;
Let&#039;s assume that you are developing an MPI application, or you are working on a complex &#039;&#039;&#039;OpenFOAM&#039;&#039;&#039; model that requires to start parallel processes over and over again just to find a bug and then fix it quickly. To do that, you can &#039;&#039;&#039;request an interactive job&#039;&#039;&#039; by adding the &#039;&#039;&#039;&amp;quot;-I&amp;quot;&#039;&#039;&#039; option to the job submission command (this is an uppercase I). Let&#039;s go to the parallel multi-node example from above:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
qsub -I -q xeon28 -l walltime=1:0:0 -l select=2:ncpus=28:mpiprocs=28:mem=60G&lt;br /&gt;
      ^    ^                  ^               ^       ^           ^      ^ ^&lt;br /&gt;
      |    |                  |               |       |           |      | + --- don&#039;t forget to specify gigabytes&lt;br /&gt;
      |    |                  |               |       |           |      + ----- the amount of memory to request per node&lt;br /&gt;
      |    |                  |               |       |           + ------------ the number of MPI tasks per nodes&lt;br /&gt;
      |    |                  |               |       + ------------------------ the number of cores per node&lt;br /&gt;
      |    |                  |               + -------------------------------- the number of nodes to select in the queue&lt;br /&gt;
      |    |                   + ----------------------------------------------- the requested time, in this case 1h&lt;br /&gt;
      |    + ------------------------------------------------------------------- the queue to be used for the job&lt;br /&gt;
      + ------------------------------------------------------------------------ request an interactive job &amp;lt;&amp;lt;===&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
When running interactive jobs with the &#039;&#039;&#039;&amp;quot;-I&amp;quot;&#039;&#039;&#039; parameter, we don&#039;t specify av job script at the end of the submission command. The interactive job will instead start (once the nodes are available) in interactive mode, meaning that the terminal session changes over from being a series of commands executed on the login server to being a series of commands being executed on the first node of the group of nodes that are allocated to the job. At this point, you can change to the desired working directory, but what you do with the allocated resources is entirely up to you. You can load modules, including MPI libraries, and then issue the commands for your application interactively and see how they execute. If you start an &#039;&#039;&#039;&amp;quot;mpirun&amp;quot;&#039;&#039;&#039;, the cores on your allocated secondary node will work as expected. There is no difference to batch mode, other than you having the ability to execute lines of commands at will.&lt;br /&gt;
&lt;br /&gt;
===Interactive Jobs with X-Windows GUI Applications===&lt;br /&gt;
&lt;br /&gt;
Interactive use can go further than that. With some of our software applications, like &#039;&#039;&#039;StarCCM+&#039;&#039;&#039;, you can run an &#039;&#039;&#039;interactive GUI application&#039;&#039;&#039; where you control the computational work from within the applications&#039; GUI. Within the GUI, you can control execution of the numerical solver that runs on as many cores as you requested, while being able to reconfigure the case through the GUI as well. Furthermore, you can visualize developing results on the fly by creating complex plots and visualizations.&lt;br /&gt;
&lt;br /&gt;
All that is need is an option &#039;&#039;&#039;&amp;quot;-X&amp;quot;&#039;&#039;&#039; being used as part of the job submission, like this:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
qsub -X -I -q xeon28 -l walltime=1:0:0 -l select=2:ncpus=28:mpiprocs=28:mem=60G&lt;br /&gt;
      ^  ^    ^                  ^               ^       ^           ^      ^ ^&lt;br /&gt;
      |  |    |                  |               |       |           |      | + --- don&#039;t forget to specify gigabytes&lt;br /&gt;
      |  |    |                  |               |       |           |      + ----- the amount of memory to request per node&lt;br /&gt;
      |  |    |                  |               |       |           + ------------ the number of MPI tasks per nodes&lt;br /&gt;
      |  |    |                  |               |       + ------------------------ the number of cores per node&lt;br /&gt;
      |  |    |                  |               + -------------------------------- the number of nodes to select in the queue&lt;br /&gt;
      |  |    |                   + ----------------------------------------------- the requested time, in this case 1h&lt;br /&gt;
      |  |    + ------------------------------------------------------------------- the queue to be used for the job&lt;br /&gt;
      |  + ------------------------------------------------------------------------ request an interactive job&lt;br /&gt;
      + --------------------------------------------------------------------------- request GUI capabilities &amp;lt;&amp;lt;===&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
===Running Multiple Jobs on Single Nodes===&lt;br /&gt;
&lt;br /&gt;
A feature that is new on ARROW is the ability to run multiple jobs on a single node. Let&#039;s assume that you are performing a sensitivity analysis on an existing model, and the model is simple enough to return results within a reasonable time on just a few cores of a higher end machine (maybe you are running SMP versions of &#039;&#039;&#039;LS-Dyna&#039;&#039;&#039;). Our high end machines have 64 cores, so lets assume you have an &#039;&#039;&#039;LS-Dyna&#039;&#039;&#039; model that runs well on 8 cores and doesn&#039;t use a whole lot of memory. In this case, you can submit individual jobs that request simply 8 cores and a fraction of the available memory available on the node, and all jobs execute independently from each other. Each job is fit into a slot where available. It is not very different from using whole nodes for everything. The important consideration is that each job is cleanly constrained into it&#039;s allotted resources using the &#039;&#039;&#039;CGROUPS&#039;&#039;&#039; functionality of modern operating systems. Because an abusive user cannot use more cores or more memory than allocated to his job, other users can safely run smaller jobs on the same node.&lt;br /&gt;
&lt;br /&gt;
Lets assume that we have a number of smaller jobs that we want to run on a single node in the &#039;&#039;&#039;&amp;quot;xeon28&amp;quot;&#039;&#039;&#039; queue. Each job would be submitted by using reduced resources that allow for sharing but that  guarantee that the jobs will be run successfully. In this case, you can &#039;&#039;&#039;submit many jobs&#039;&#039;&#039; in the following manner (with a job script for the small jobs, each of which can &#039;&#039;&#039;request varying resources&#039;&#039;&#039; if needed - some may want to run on 5 cores, others on 3):&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
#!/bin/bash&lt;br /&gt;
#&lt;br /&gt;
# the following ensures that you will change into the directory where you are&lt;br /&gt;
# submitting the job from.&lt;br /&gt;
cd $PBS_O_WORKDIR&lt;br /&gt;
#&lt;br /&gt;
# now we sleep for 300 seconds and waste time. This is a placeholder for your application,&lt;br /&gt;
# which would be doing useful work for you.&lt;br /&gt;
sleep 300&lt;br /&gt;
#&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
Now we submit a variety of these jobs (11 total in this example) to the &#039;&#039;&#039;&amp;quot;xeon28&amp;quot;&#039;&#039;&#039; queue for execution (note that the first few jobs request different amounts of memory and core counts):&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
qsub -q xeon28 -l walltime=1:0:0 -l select=1:ncpus=12:mpiprocs=12:mem=5G small.job &lt;br /&gt;
qsub -q xeon28 -l walltime=1:0:0 -l select=1:ncpus=10:mpiprocs=10:mem=7G small.job &lt;br /&gt;
qsub -q xeon28 -l walltime=1:0:0 -l select=1:ncpus=8:mpiprocs=8:mem=9G small.job &lt;br /&gt;
qsub -q xeon28 -l walltime=1:0:0 -l select=1:ncpus=16:mpiprocs=16:mem=20G small.job &lt;br /&gt;
qsub -q xeon28 -l walltime=1:0:0 -l select=1:ncpus=2:mpiprocs=2:mem=2G small.job &lt;br /&gt;
qsub -q xeon28 -l walltime=1:0:0 -l select=1:ncpus=2:mpiprocs=2:mem=2G small.job &lt;br /&gt;
qsub -q xeon28 -l walltime=1:0:0 -l select=1:ncpus=2:mpiprocs=2:mem=2G small.job &lt;br /&gt;
qsub -q xeon28 -l walltime=1:0:0 -l select=1:ncpus=2:mpiprocs=2:mem=2G small.job &lt;br /&gt;
qsub -q xeon28 -l walltime=1:0:0 -l select=1:ncpus=2:mpiprocs=2:mem=2G small.job &lt;br /&gt;
qsub -q xeon28 -l walltime=1:0:0 -l select=1:ncpus=2:mpiprocs=2:mem=2G small.job &lt;br /&gt;
qsub -q xeon28 -l walltime=1:0:0 -l select=1:ncpus=2:mpiprocs=2:mem=2G small.job &lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
They are now running in the order of submission, allocated on as few nodes in the &amp;quot;xeon28&amp;quot; queue as necessary. Only 2 nodes are being loaded quite heavily, and 4 more cores are in use on a third node.&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
Nov 20 23:34 ley@login3:myjobfolder$ qstat -an1&lt;br /&gt;
&lt;br /&gt;
pbs: &lt;br /&gt;
                                                            Req&#039;d  Req&#039;d   Elap&lt;br /&gt;
Job ID          Username Queue    Jobname    SessID NDS TSK Memory Time  S Time&lt;br /&gt;
--------------- -------- -------- ---------- ------ --- --- ------ ----- - -----&lt;br /&gt;
&lt;br /&gt;
3082.pbs        ley      xeon28   small.job  813221   1  12    5gb 01:00 R 00:01 p001/0*12&lt;br /&gt;
3083.pbs        ley      xeon28   small.job  813288   1  10    7gb 01:00 R 00:01 p001/1*10&lt;br /&gt;
3084.pbs        ley      xeon28   small.job  671792   1   8    9gb 01:00 R 00:01 p002/0*8&lt;br /&gt;
3085.pbs        ley      xeon28   small.job  671845   1  16   20gb 01:00 R 00:01 p002/1*16&lt;br /&gt;
3086.pbs        ley      xeon28   small.job  813361   1   2    2gb 01:00 R 00:00 p001/2*2&lt;br /&gt;
3087.pbs        ley      xeon28   small.job  813413   1   2    2gb 01:00 R 00:00 p001/3*2&lt;br /&gt;
3088.pbs        ley      xeon28   small.job  813464   1   2    2gb 01:00 R 00:00 p001/4*2&lt;br /&gt;
3089.pbs        ley      xeon28   small.job  671912   1   2    2gb 01:00 R 00:00 p002/2*2&lt;br /&gt;
3090.pbs        ley      xeon28   small.job  671969   1   2    2gb 01:00 R 00:00 p002/3*2&lt;br /&gt;
3091.pbs        ley      xeon28   small.job  632092   1   2    2gb 01:00 R 00:00 p003/0*2&lt;br /&gt;
3092.pbs        ley      xeon28   small.job  632100   1   2    2gb 01:00 R 00:00 p003/1*2&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
This is a particularly effective strategy to run concurrently many cases that don&#039;t scale well beyond a few cores. When running them on fewer cores but many of them at the same time, the overall processing rate will be much higher than executing them the traditional way.&lt;br /&gt;
&lt;br /&gt;
===Running Jobs using GPUs===&lt;br /&gt;
&lt;br /&gt;
The principle of running multiple jobs on a single node becomes particularly important when using servers equipped with &#039;&#039;&#039;GPUs&#039;&#039;&#039; for &#039;&#039;&#039;ML/AI&#039;&#039;&#039; applications. The cluster doesn&#039;t have a whole lot of &#039;&#039;&#039;GPUs&#039;&#039;&#039; at this point. We have three machines with three &#039;&#039;&#039;A4000&#039;&#039;&#039; GOUs, a &#039;&#039;&#039;total of 9 A4000 GPUs&#039;&#039;&#039;. Then we have a much more powerful single machine with our &#039;&#039;&#039;four A6000 GPUs&#039;&#039;&#039;.&lt;br /&gt;
&lt;br /&gt;
Using multiple GPUs in a single application is still something where the software has to be designed with hardware in mind. GPUs have several methods of communicating with each other, e.g. very fast &#039;&#039;&#039;NVLINK&#039;&#039;&#039; between pairs of GPUs, GPUs being directly connected to one of the two CPUs in the system and thus being able to communicate faster, and GPUs that have to jump between processors when communicating, and then the whole issue of having to go possibly through PCIe bridges.&lt;br /&gt;
&lt;br /&gt;
On our system, we are providing the ability to &#039;&#039;&#039;work mostly with individual GPUs&#039;&#039;&#039;. Users can also reserve entire nodes and develop or run applications that are adapted to that hardware, including several GPUs installed on that node. One thing we do not provide is the ability of GPU to GPU communication between nodes. Thus, a job cannot request more than one GPU node at a time.&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
qsub -q a4000 -I -l walltime=1:0:0 -l select=1:ncpus=5:mem=150G:ngpus=1&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
With these specifications, three single GPU jobs can run on a single server. Each job sees only one of the reserved GPU.&lt;br /&gt;
&lt;br /&gt;
To run a massive GPU job on 64 cores with 4 &#039;&#039;&#039;A6000 GPUs&#039;&#039;&#039;, submit the job like this:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
qsub -q a6000 -I -l walltime=1:0:0 -l select=1:ncpus=64:mem=725G:ngpus=4&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
===Manual pages for qsub===&lt;br /&gt;
&lt;br /&gt;
To learn more about the &amp;quot;qsub&amp;quot; command, you can use the command &amp;quot;man qsub&amp;quot;, which will print a lot of detailed information about the capabilities of this command.&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
$ man qsub&lt;br /&gt;
&lt;br /&gt;
qsub(1B)                                               PBS Professional                                              qsub(1B)&lt;br /&gt;
&lt;br /&gt;
NAME&lt;br /&gt;
       qsub - submit a job to PBS&lt;br /&gt;
&lt;br /&gt;
SYNOPSIS&lt;br /&gt;
       qsub [-a &amp;lt;date and time&amp;gt;] [-A &amp;lt;account string&amp;gt;] [-c &amp;lt;checkpoint spec&amp;gt;]&lt;br /&gt;
            [-C &amp;lt;directive prefix&amp;gt;] [-e &amp;lt;path&amp;gt;] [-f] [-h]&lt;br /&gt;
            [-I [-G [-- &amp;lt;GUI application/script&amp;gt;]] | [-X]] [-j &amp;lt;join&amp;gt;]&lt;br /&gt;
            [-J &amp;lt;range&amp;gt; [%&amp;lt;max subjobs]] [-k &amp;lt;discard&amp;gt;] [-l &amp;lt;resource list&amp;gt;]&lt;br /&gt;
            [-m &amp;lt;mail events&amp;gt;] [-M &amp;lt;user list&amp;gt;] [-N &amp;lt;name&amp;gt;] [-o &amp;lt;path&amp;gt;]&lt;br /&gt;
            [-p &amp;lt;priority&amp;gt;] [-P &amp;lt;project&amp;gt;] [-q &amp;lt;destination&amp;gt;] [-r &amp;lt;y|n&amp;gt;]&lt;br /&gt;
            [-R &amp;lt;remove options&amp;gt;] [-S &amp;lt;path list&amp;gt;] [-u &amp;lt;user list&amp;gt;]&lt;br /&gt;
            [-v &amp;lt;variable list&amp;gt;] [-V] [-W &amp;lt;additional attributes&amp;gt;] [-z]&lt;br /&gt;
            [- | &amp;lt;script&amp;gt; | -- &amp;lt;executable&amp;gt; [&amp;lt;arguments to executable&amp;gt;]]&lt;br /&gt;
       qsub --version&lt;br /&gt;
&lt;br /&gt;
DESCRIPTION&lt;br /&gt;
       You use the qsub command to submit a batch job to PBS.  Submitting a PBS job specifies a task, requests resources, and&lt;br /&gt;
       sets job attributes.&lt;br /&gt;
... &amp;lt;many more pages&amp;gt;&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
==LS-Dyna on the ARROW Cluster==&lt;br /&gt;
&lt;br /&gt;
===Currently Available LS-Dyna Versions===&lt;br /&gt;
&lt;br /&gt;
The following is a list of &#039;&#039;&#039;LS-Dyna versions&#039;&#039;&#039; available on &#039;&#039;&#039;ARROW&#039;&#039;&#039; after the latest reconfiguration of the system. As per LSTC/ANSYS, &#039;&#039;&#039;versions before 14.0.0 are not necessarily fully supported any longer&#039;&#039;&#039; because they are supposedly not compatible with modern operating systems and cannot be made to work reliably. We have tested the listed older versions of LS-Dyna and they have passed basic tests. They may not behave exactly as they did on the old CentOS 7 operating system, and time will show whether they can still be used or whether you will need to update your models and use a fully supported version.&lt;br /&gt;
&lt;br /&gt;
All versions are loaded using the &#039;&#039;&#039;&amp;quot;module load&amp;quot;&#039;&#039;&#039; command. Versions can be listed with the &#039;&#039;&#039;&amp;quot;module avail ls-dyna&amp;quot;&#039;&#039;&#039; command. To load one of the modules, use the following syntax:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
module load ls-dyna/14.2.0/mpi-d8-ifort190-avx512&lt;br /&gt;
            ^       ^      ^   ^  ^        ^&lt;br /&gt;
            |       |      |   |  |        + --- specify the extended instruction set needed for execution&lt;br /&gt;
            |       |      |   |  + ------------ load the version of the compiler that was used to create this &lt;br /&gt;
            |       |      |   + --------------- load the version that supports double precision variables&lt;br /&gt;
            |       |      + ------------------- load the MPP (MPI) version of LS-Dyna&lt;br /&gt;
            |       + -------------------------- load specifically version 14.2.0&lt;br /&gt;
            + ---------------------------------- load a version of LS-Dyna&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
The version string is composed of multiple elements to indicate variants in compilers and compiler options. Use the following guideline to choose an appropriate version to load:&lt;br /&gt;
&lt;br /&gt;
* &#039;&#039;&#039;&amp;quot;1&amp;quot;&#039;&#039;&#039; or &#039;&#039;&#039;&amp;quot;mpi&amp;quot;&#039;&#039;&#039; indicates whether this is a single node version of LS-Dyna (&#039;&#039;&#039;SMP&#039;&#039;&#039;) or whether this is a multi-node MPI version (&#039;&#039;&#039;MPP&#039;&#039;&#039;). All MPI versions use the &#039;&#039;&#039;IntelMPI 2022&#039;&#039;&#039; libraries which have been tested thoroughly on ARROW. MPI versions will use the Infiniband Network of ARROW for high-speed and low-latency inter-process communication using RDMA (remote direct memory access).&lt;br /&gt;
* All LS-Dyna versions are available in either &#039;&#039;&#039;floating point&#039;&#039;&#039; or &#039;&#039;&#039;double precision variants&#039;&#039;&#039;. Floating point variants use 4 bytes to represent a value, and double precision variants use 8 bytes. There are pros and cons for choosing one over the other variant. With regards to computational efficiency, both perform nearly the same because all machines are equipped with 64-bit CPUs.&lt;br /&gt;
** &#039;&#039;&#039;&amp;quot;f4&amp;quot;&#039;&#039;&#039; floating point versions&lt;br /&gt;
*** Pros: These require significantly less memory to run. Results occupy less disk space, and can be transferred significantly faster into and out of ARROW.&lt;br /&gt;
*** Cons: The numerical resolution is limited to 7 significant digits, which is often undesirable when dealing with mathematical operations on small and large numbers at the same time.&lt;br /&gt;
** &#039;&#039;&#039;&amp;quot;r8&amp;quot;&#039;&#039;&#039; double precision versions&lt;br /&gt;
*** Pros: The numerical resolution is about twice the number of significant digits compare to &amp;quot;f4&amp;quot;, which helps when when dealing with mathematical operations on small and large numbers at the same time.&lt;br /&gt;
*** Cons: These require more memory to run. Results occupy more disk space, and it takes longer to transfer data into and out of ARROW.&lt;br /&gt;
* There are two more identifiers to choose from when it comes to the variants of the executables: the specific compiler used to create the executable and the specific processor instruction set required for running the executable.&lt;br /&gt;
** For modern versions of LS-Dyna, two compilers have been used by the developers to create LS-Dyna executables: the &#039;&#039;&#039;Intel Fortran Compiler&#039;&#039;&#039; and the &#039;&#039;&#039;AOCC (AMD Optimizing C/C++ and Fortran) compiler&#039;&#039;&#039;. Both variants of the software are supported on ARROW. This gives users the opportunity to choose an alternate variant of the same LS-Dyna version when running into bugs or crashes.&lt;br /&gt;
** The variants based on the various instruction set extensions (&#039;&#039;&#039;SSE2&#039;&#039;&#039;, &#039;&#039;&#039;AVX2&#039;&#039;&#039;, &#039;&#039;&#039;AVX512&#039;&#039;&#039;, and so on) gives users even more options when choosing an alternate LS-Dyna variant of the same version when running into bugs or crashes. These instruction sets are mostly related to performance gains on specific processors. We have not performed thorough performance tests and cannot recommend specific versions right now.&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
$ module avail ls-dyna&lt;br /&gt;
--------------------------------------------- /shared/apps/modulefiles ---------------------------------------------&lt;br /&gt;
ls-dyna/09.3.1/1-d8-ifort131           ls-dyna/12.2.1/mpi-f4-ifort160-sse2    ls-dyna/14.2.0/mpi-f4-ifort190-avx512  &lt;br /&gt;
ls-dyna/09.3.1/1-f4-ifort131           ls-dyna/12.2.2/1-d8-ifort160-sse2      ls-dyna/14.2.0/mpi-f4-ifort190-sse2    &lt;br /&gt;
ls-dyna/09.3.1/mpi-d8-ifort131-avx2    ls-dyna/12.2.2/1-f4-ifort160-sse2      ls-dyna/15.0.2/1-d8-ifort190-sse2      &lt;br /&gt;
ls-dyna/09.3.1/mpi-d8-ifort131-avx512  ls-dyna/12.2.2/mpi-d8-aocc400-avx2     ls-dyna/15.0.2/1-f4-ifort190-sse2      &lt;br /&gt;
ls-dyna/09.3.1/mpi-f4-ifort131-avx2    ls-dyna/12.2.2/mpi-d8-ifort160-avx2    ls-dyna/15.0.2/mpi-d8-aocc400-avx2     &lt;br /&gt;
ls-dyna/09.3.1/mpi-f4-ifort131-avx512  ls-dyna/12.2.2/mpi-d8-ifort160-sse2    ls-dyna/15.0.2/mpi-d8-ifort190-avx2    &lt;br /&gt;
ls-dyna/10.2.0/1-d8-ifort160           ls-dyna/12.2.2/mpi-f4-aocc400-avx2     ls-dyna/15.0.2/mpi-d8-ifort190-avx512  &lt;br /&gt;
ls-dyna/10.2.0/1-f4-ifort160           ls-dyna/12.2.2/mpi-f4-ifort160-avx2    ls-dyna/15.0.2/mpi-d8-ifort190-sse2    &lt;br /&gt;
ls-dyna/11.0.0/1-d8-ifort160           ls-dyna/12.2.2/mpi-f4-ifort160-sse2    ls-dyna/15.0.2/mpi-f4-aocc400-avx2     &lt;br /&gt;
ls-dyna/11.0.0/1-f4-ifort160           ls-dyna/13.0.0/1-d8-ifort190           ls-dyna/15.0.2/mpi-f4-ifort190-avx2    &lt;br /&gt;
ls-dyna/11.1.0/1-d8-ifort160-sse2      ls-dyna/13.0.0/1-f4-ifort190           ls-dyna/15.0.2/mpi-f4-ifort190-avx512  &lt;br /&gt;
ls-dyna/11.1.0/1-f4-ifort160-sse2      ls-dyna/13.0.0/mpi-d8-ifort190-avx2    ls-dyna/15.0.2/mpi-f4-ifort190-sse2    &lt;br /&gt;
ls-dyna/11.2.0/1-d8-ifort160           ls-dyna/13.0.0/mpi-d8-ifort190-sse2    ls-dyna/16.0.0/1-d8-aocc420-avx2       &lt;br /&gt;
ls-dyna/11.2.0/1-f4-ifort160           ls-dyna/13.0.0/mpi-f4-ifort190-avx2    ls-dyna/16.0.0/1-d8-aocc420-avx512     &lt;br /&gt;
ls-dyna/11.2.0/mpi-f4-ifort160-avx2    ls-dyna/13.0.0/mpi-f4-ifort190-sse2    ls-dyna/16.0.0/1-d8-ifort190-sse2      &lt;br /&gt;
ls-dyna/11.2.0/mpi-f4-ifort160-sse2    ls-dyna/13.1.0/mpi-d8-aocc310-avx2     ls-dyna/16.0.0/1-f4-aocc420-avx2       &lt;br /&gt;
ls-dyna/11.2.1/1-d8-ifort160           ls-dyna/13.1.0/mpi-d8-ifort190-avx2    ls-dyna/16.0.0/1-f4-aocc420-avx512     &lt;br /&gt;
ls-dyna/11.2.1/1-f4-ifort160           ls-dyna/13.1.0/mpi-d8-ifort190-sse2    ls-dyna/16.0.0/1-f4-ifort190-sse2      &lt;br /&gt;
ls-dyna/11.2.1/mpi-d8-ifort160-avx2    ls-dyna/13.1.0/mpi-f4-aocc310-avx2     ls-dyna/16.0.0/mpi-d8-aocc420-avx2     &lt;br /&gt;
ls-dyna/11.2.1/mpi-d8-ifort160-sse2    ls-dyna/13.1.0/mpi-f4-ifort190-avx2    ls-dyna/16.0.0/mpi-d8-aocc420-avx512   &lt;br /&gt;
ls-dyna/11.2.1/mpi-f4-ifort160-avx2    ls-dyna/13.1.0/mpi-f4-ifort190-sse2    ls-dyna/16.0.0/mpi-d8-ifort190-avx2    &lt;br /&gt;
ls-dyna/11.2.1/mpi-f4-ifort160-sse2    ls-dyna/13.1.1/mpi-d8-ifort190-avx2    ls-dyna/16.0.0/mpi-d8-ifort190-avx512  &lt;br /&gt;
ls-dyna/11.2.2/mpi-d8-ifort160-avx2    ls-dyna/13.1.1/mpi-d8-ifort190-sse2    ls-dyna/16.0.0/mpi-d8-ifort190-sse2    &lt;br /&gt;
ls-dyna/11.2.2/mpi-d8-ifort160-sse2    ls-dyna/13.1.1/mpi-f4-ifort190-avx2    ls-dyna/16.0.0/mpi-f4-aocc420-avx2     &lt;br /&gt;
ls-dyna/11.2.2/mpi-f4-ifort160-avx2    ls-dyna/13.1.1/mpi-f4-ifort190-sse2    ls-dyna/16.0.0/mpi-f4-aocc420-avx512   &lt;br /&gt;
ls-dyna/11.2.2/mpi-f4-ifort160-sse2    ls-dyna/14.0.0/1-d8-ifort190           ls-dyna/16.0.0/mpi-f4-ifort190-avx2    &lt;br /&gt;
ls-dyna/12.1.0/1-d8-ifort160           ls-dyna/14.0.0/1-f4-ifort190           ls-dyna/16.0.0/mpi-f4-ifort190-avx512  &lt;br /&gt;
ls-dyna/12.1.0/1-f4-aocc310            ls-dyna/14.0.0/mpi-d8-aocc310-avx2     ls-dyna/16.0.0/mpi-f4-ifort190-sse2    &lt;br /&gt;
ls-dyna/12.1.0/1-f4-ifort160           ls-dyna/14.0.0/mpi-d8-ifort190-avx2    ls-dyna/16.1.0/mpi-d8-aocc420-avx2     &lt;br /&gt;
ls-dyna/12.1.0/mpi-d8-aocc310-avx2     ls-dyna/14.0.0/mpi-d8-ifort190-sse2    ls-dyna/16.1.0/mpi-d8-aocc420-avx512   &lt;br /&gt;
ls-dyna/12.1.0/mpi-d8-ifort160-avx2    ls-dyna/14.0.0/mpi-f4-ifort190-avx2    ls-dyna/16.1.0/mpi-d8-ifort190-avx2    &lt;br /&gt;
ls-dyna/12.1.0/mpi-d8-ifort160-sse2    ls-dyna/14.0.0/mpi-f4-ifort190-sse2    ls-dyna/16.1.0/mpi-d8-ifort190-avx512  &lt;br /&gt;
ls-dyna/12.1.0/mpi-f4-aocc310-avx2     ls-dyna/14.1.0/1-d8-ifort190-sse2      ls-dyna/16.1.0/mpi-d8-ifort190-sse2    &lt;br /&gt;
ls-dyna/12.1.0/mpi-f4-ifort160-avx2    ls-dyna/14.1.0/1-f4-ifort190-sse2      ls-dyna/16.1.0/mpi-f4-aocc420-avx2     &lt;br /&gt;
ls-dyna/12.1.0/mpi-f4-ifort160-sse2    ls-dyna/14.1.0/mpi-d8-aocc400-avx2     ls-dyna/16.1.0/mpi-f4-aocc420-avx512&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
===Submitting an LS-Dyna Job===&lt;br /&gt;
&lt;br /&gt;
&amp;lt;BLOCKQUOTE&amp;gt;&lt;br /&gt;
[[File:Attention.jpg|25px]] &#039;&#039;&#039;IMPORTANT NOTE:&#039;&#039;&#039; The job/queue manager can now track the number of LS-Dyna licenses given out to individual&lt;br /&gt;
jobs. At submission time, it is not possible to know what software a user may run. But by adding the clause &amp;quot;-l dynalic&amp;quot; at submission time,&lt;br /&gt;
the queue manager can calculate the total number of cores required and keep track of LS-Dyna licenses used by the job. When loading a version of LS-Dyna, a check will be performed, and LS-Dyna will be prevented from running if the &amp;quot;-l dynalic&amp;quot; clause was not used when submitting the job.&#039;&#039;&lt;br /&gt;
&lt;br /&gt;
&amp;lt;!--&lt;br /&gt;
&#039;&#039;The job/queue manager can track the number of LS-Dyna licenses to some degree. If all &#039;&#039;&#039;LS-Dyna users&#039;&#039;&#039; cooperate and use a script like the one shown below when submitting their jobs, the total number of concurrent &#039;&#039;&#039;LS-Dyna licenses&#039;&#039;&#039; will be tracked by the job manager correctly. That means that users can submit any number of LS-Dyna jobs, and jobs will only start when a sufficient number of licenses is available. This is managed by the &#039;&#039;&#039;&amp;quot;dynalic&amp;quot;&#039;&#039;&#039; resource at the end of the select statement. In this example, a 2-node job on 64-core nodes will need a total of &#039;&#039;&#039;&amp;quot;dynalic=128&amp;quot;&#039;&#039;&#039; licenses. This accounting breaks down when users don&#039;t use the &#039;&#039;&#039;&amp;quot;dynalic=XXX&amp;quot;&#039;&#039;&#039; statement, or when they don&#039;t calculate the number of licenses correctly. In that case, LS-Dyna jobs of all users are subject to sudden failure when LS-Dyna licenses run out. Please understand the importance of this specific setting in your job.&#039;&#039;&lt;br /&gt;
--&amp;gt;&lt;br /&gt;
&amp;lt;/BLOCKQUOTE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
Furthermore, careful consideration should be given with regards to choice of resources for an &#039;&#039;&#039;LS-Dyna job&#039;&#039;&#039;. With 64 cores available on a single node in the &#039;&#039;&#039;&amp;quot;epyc1&amp;quot;&#039;&#039;&#039; and &#039;&#039;&#039;&amp;quot;epyc2&amp;quot;&#039;&#039;&#039; queues, it may be counterproductive to run a job on two nodes instead of a single node. Users should run their jobs with different numbers of nodes and determine whether performance increases. It may well decrease when running a job on two or more nodes. The outcome of such tests will tell what the best allocation of resources will be.&lt;br /&gt;
&lt;br /&gt;
Most users use a job script like the following. All methods for job submission the the previous chapters apply as well, so there is a lot of flexibility:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
#!/bin/bash&lt;br /&gt;
#&lt;br /&gt;
#PBS -q epyc1&lt;br /&gt;
#PBS -l walltime=12:0:0&lt;br /&gt;
#PBS -l select=2:ncpus=64:mpiprocs=64:mem=225G,dynalic&lt;br /&gt;
#PBS -N JobName&lt;br /&gt;
#PBS -e log.error&lt;br /&gt;
#PBS -o log.output&lt;br /&gt;
#&lt;br /&gt;
cd $PBS_O_WORKDIR&lt;br /&gt;
#&lt;br /&gt;
module load ls-dyna/12.2.1/mpi-f4-ifort160-avx2&lt;br /&gt;
module load dynamore/current&lt;br /&gt;
#&lt;br /&gt;
mpirun ls-dyna i=main.k memory1=300m memory2=100m &amp;gt; dyna.log&lt;br /&gt;
#&lt;br /&gt;
# when using the Dynamore tools, you can start something like this at the end&lt;br /&gt;
DM.plotcprs.lnx -merge &amp;gt;&amp;gt; dyna.log&lt;br /&gt;
#&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
===LSTC Tools: LS-OPT and LS-PREPOST===&lt;br /&gt;
&lt;br /&gt;
For the new Rocky 9 cluster, I have not looked deeply into the ls-opt and ls-prepost packages that were installed. I noticed though that the LSTC server provided access to much newer versions of both software packages. If you would like to learn more or have a specific version in mind, I can easily download and install it for you.&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
$ module avail ls-opt&lt;br /&gt;
----------------------------------------------- /shared/apps/modulefiles ------------------------------------------------&lt;br /&gt;
ls-opt/5.1.1  ls-opt/6.0.0  ls-opt/7.0.0  ls-opt/7.0.2   ls-opt/2022R2  &lt;br /&gt;
ls-opt/5.2.1  ls-opt/6.1.0  ls-opt/7.0.1  ls-opt/2022R1  ls-opt/2023R1  &lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
To start the software, type:&lt;br /&gt;
 lsopt&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
$ module avail ls-prepost&lt;br /&gt;
----------------------------------------------- /shared/apps/modulefiles ------------------------------------------------&lt;br /&gt;
ls-prepost/4.5.10  ls-prepost/4.8.13  ls-prepost/4.8.30  ls-prepost/4.9.16  ls-prepost/4.10.7 &lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
To start the software, type:&lt;br /&gt;
 lsprepost&lt;br /&gt;
&lt;br /&gt;
===Dynamore Software===&lt;br /&gt;
&lt;br /&gt;
The Dynamore tools are available as a module:&lt;br /&gt;
&lt;br /&gt;
 module load dynamore/current&lt;br /&gt;
&lt;br /&gt;
We typically acquire a yearly license for the tools as we purchase licenses for LS-Dyna.&lt;br /&gt;
&lt;br /&gt;
===Vendor License File Installation===&lt;br /&gt;
&lt;br /&gt;
If you would like for us to install a vendor license for LS-Dyna models, please contact us for the required information. We can send you the general LS-Dyna license file to show the host ids for the license server. Using that information, your vendor should be able to create a vendor license file. Please send that file to us per Email or by other means.&lt;br /&gt;
&lt;br /&gt;
==StarCCM+ on the ARROW Cluster==&lt;br /&gt;
&lt;br /&gt;
===Currently Available StarCCM+ Versions===&lt;br /&gt;
&lt;br /&gt;
As of late 2025, we have the following versions of &#039;&#039;&#039;StarCCM+&#039;&#039;&#039; available on the cluster:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
$ module avail starccm&lt;br /&gt;
---------------------------- /shared/apps/modulefiles ----------------------------&lt;br /&gt;
starccm/15.02.007-R8  starccm/16.06.008-R8  starccm/18.06.006-R8  &lt;br /&gt;
starccm/15.02.009-R8  starccm/17.02.007-R8  starccm/19.02.009-R8  &lt;br /&gt;
starccm/15.04.008-R8  starccm/17.02.008-R8  starccm/20.04.007-R8  &lt;br /&gt;
starccm/15.06.008-R8  starccm/17.04.007-R8  starccm/20.06.007-R8  &lt;br /&gt;
starccm/16.02.008-R8  starccm/17.06.007-R8  &lt;br /&gt;
starccm/16.04.007-R8  starccm/18.04.008-R8&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
If using a &#039;&#039;&#039;single node&#039;&#039;&#039; for StarCCM+, job submission (for an interactive job) is simple and will use appropriate default settings:&lt;br /&gt;
&lt;br /&gt;
 qsub -I -X -q epyc1 -l walltime=20:00:00&lt;br /&gt;
&lt;br /&gt;
StarCCM+ can make use of the job scheduler attributes by automatically obtaining the number of cores and other resources from OpenPBS. In this case, the default number of cores and mpi processes for StarCCM+ are both 64 when using the epyc1 queue. So you can start your StarCCM+ run with:&lt;br /&gt;
&lt;br /&gt;
 module load starccm/15.02.007-R8 (or any other version)&lt;br /&gt;
 starccm+ -bs pbs&lt;br /&gt;
&lt;br /&gt;
In this case, there is no need for StarCCM+ to be told to run the case in parallel with the selected number of cores/mpiprocs.&lt;br /&gt;
&lt;br /&gt;
This can get a bit more complex when running on multiple nodes or when requesting high memory nodes. In that case you would use job submission parameters as shown below:&lt;br /&gt;
&lt;br /&gt;
 qsub -I -X -q epyc1 -l walltime=20:00:00,select=2:ncpus=64:mpiprocs=61:mem=500GB&lt;br /&gt;
&lt;br /&gt;
Requesting nodes that can satisfy those resources, two nodes with these attributes must exist. We have multiple nodes with 512GB in the epyc1 queue, meaning that this job will run on two machines that have at least the required amount of memory installed (on each node). The job will be queued until two machines like this will be available. If no machines with these resources exist, the job will stay in the queue forever. Therefore, you have to craft the submission string carefully.&lt;br /&gt;
&lt;br /&gt;
To accommodate high memory jobs, the nodes have been assigned priorities for assignment. Low memory jobs have the highest priority and will be assigned to nodes that can accommodate the request. High memory nodes have the lowest priority, meaning that they are the last ones given out to users. This makes it more likely that a high memory job can be run soon when the cluster is moderately loaded with jobs.&lt;br /&gt;
&lt;br /&gt;
StarCCM+ will always use the Intel MPI fabric. Other MPI versions do not work, even when selected on the command line.&lt;br /&gt;
&lt;br /&gt;
==OpenFOAM on the ARROW Cluster==&lt;br /&gt;
&lt;br /&gt;
===Currently Available OpenFOAM  Versions===&lt;br /&gt;
&lt;br /&gt;
As of late 2025, we have the following versions of OpenFOAM available on the cluster:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
$ module avail openfoam&lt;br /&gt;
------------ /shared/apps/modulefiles ------------&lt;br /&gt;
openfoam/9   openfoam/13      openfoam/v2312  &lt;br /&gt;
openfoam/10  openfoam/13-amd  openfoam/v2406  &lt;br /&gt;
openfoam/11  openfoam/v2212   &lt;br /&gt;
openfoam/12  openfoam/v2306&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
Contact us if you encounter problems; there can be various reasons why OpenFOAM may have trouble on certain hardware or when compiling dynamic code. When loading OpenFOAM modules, a number of dependencies will be automatically loaded for you, and you don&#039;t have to load those yourself. For example:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
$ module load openfoam/13&lt;br /&gt;
Loading openfoam/13&lt;br /&gt;
  Loading requirement: intel/2024.2.0/mpi/2021.13 gcc/gcc-12.1.0&lt;br /&gt;
&lt;br /&gt;
$ module list&lt;br /&gt;
Currently Loaded Modulefiles:&lt;br /&gt;
 1) intel/2024.2.0/mpi/2021.13   2) gcc/gcc-12.1.0   3) openfoam/13&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
In this case, OpenFOAM 13 loads the Intel 2024 MPI module, and loads the GCC compiler 12.1. OpenFOAM was compiled from source, and has been compiled specifically with that compiler and MPI version, so it make little sense to use other compilers or MPI libraries.&lt;br /&gt;
&lt;br /&gt;
Note: We have found a problem with running the Intel 2024 MPI library in the amd64 queue. Therefore, we have a modified module that uses the Intel 2022 library (I know -- Intel 2022 gives you the 2021 MPI libraries, but that is the way Intel distributes this software):&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
$ module load openfoam/13-amd &lt;br /&gt;
Loading mpi version 2021.7.0&lt;br /&gt;
Loading openfoam/13-amd&lt;br /&gt;
  Loading requirement: intel/2022.2.0/mpi/2021.7.0 gcc/gcc-12.1.0&lt;br /&gt;
&lt;br /&gt;
$ module list&lt;br /&gt;
Currently Loaded Modulefiles:&lt;br /&gt;
 1) intel/2022.2.0/mpi/2021.7.0   2) gcc/gcc-12.1.0   3) openfoam/13-amd&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
If you are compiling OpenFOAM yourself, the modules are of little help. You would need to select the appropriate MPI version and compiler before doing so, and then consistently load them before running your OpenFOAM executables. Within the &amp;quot;etc/bashrc&amp;quot; file in the source code tree, you want to set the MPI library to INTELMPI. As usual with self-compiled versions of OpenFOAM, you would &amp;quot;source etc/bashrc&amp;quot; to set up your personal environment to run your home-brew version of OpenFOAM. Contact us if you need to learn more about compiling OpenFOAM on the system.&lt;br /&gt;
&lt;br /&gt;
==Additional Software Applications and Libraries==&lt;br /&gt;
&lt;br /&gt;
===Loadable GCC Compiler Versions===&lt;br /&gt;
&lt;br /&gt;
The Rocky 9.6 operating system uses the GCC 11.5 compiler. That should be sufficient for most users when compiling your own applications. In case you need to use either a more up-to-date compiler, or if you need an older compiler for compatibility, we make the following versions available as loadable modules.&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
$ module avail gcc&lt;br /&gt;
------------ /shared/apps/modulefiles ------------&lt;br /&gt;
gcc/gcc-4.9.4  gcc/gcc-7.5.0  gcc/gcc-10.3.0  &lt;br /&gt;
gcc/gcc-5.5.0  gcc/gcc-8.5.0  gcc/gcc-11.3.0  &lt;br /&gt;
gcc/gcc-6.5.0  gcc/gcc-9.5.0  gcc/gcc-12.1.0&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
Additional versions can be created and made available as modules as well. If you need a specific version that is not currently available, please ask us to compiler and install it. If necessary, we may be able to provide access to other compilers, for example LLVM. We do not provide access to proprietary compilers at this time.&lt;br /&gt;
&lt;br /&gt;
===MPI Libraries and Runtimes===&lt;br /&gt;
&lt;br /&gt;
While we seem to have a variety of MPI versions and flavors available to users, the only MPI versions that allow us to run software over Infiniband are the Intel MPI libraries. Some of the installed alternatives are likely to fail, or will have a set of environment variables that have to be set. All major engineering software packages that we offer are pre-configured with specific MPI versions and settings that have been tested and/or provided by the vendors.&lt;br /&gt;
&lt;br /&gt;
Note: Some MPI libraries may seem to work. They may allow your MPI application to run. But inter-process network communication may travel through the rather slow and high-latency Ethernet fabric, making MPI applications very ineffective and are probably not worth while.&lt;br /&gt;
&lt;br /&gt;
===MatLab Runtimes===&lt;br /&gt;
&lt;br /&gt;
We can install MatLAB run time libraries as needed and have them available as loadable modules. Recently, we had a problem with MatLAB run time libraries being identified as security vulnerabilities. Contact us if you need them installed for one of your projects.&lt;br /&gt;
&lt;br /&gt;
===Anaconda and variants (miniconda etc)===&lt;br /&gt;
&lt;br /&gt;
Our current practice is to have users download and install their own versions of Anaconda and its variants in their own home directories. This allows for maximum flexibility when it comes to installable software modules, and users can maintain the installation, upgrades, and maintenance themselves. If you encounter issues, please contact us. One known side effect of Anaconda installations is a performance hit when starting your software, e.g. python scripts may take 30 seconds or more to execute. This is an artefact caused by the Lustre file system, which has been designed for large files accessible from many machines simultaneously. Performance on reading many small files has not been considered and is fairly poor. Again, contact us and we will design a solution for you as needed.&lt;/div&gt;</summary>
		<author><name>Ley</name></author>
	</entry>
	<entry>
		<id>https://wiki.anl.gov/wiki_tracc/index.php?title=Job_Submission_and_Monitoring&amp;diff=2491</id>
		<title>Job Submission and Monitoring</title>
		<link rel="alternate" type="text/html" href="https://wiki.anl.gov/wiki_tracc/index.php?title=Job_Submission_and_Monitoring&amp;diff=2491"/>
		<updated>2026-02-26T18:14:34Z</updated>

		<summary type="html">&lt;p&gt;Ley: /* Resource Summary View */&lt;/p&gt;
&lt;hr /&gt;
&lt;div&gt;{| align=&amp;quot;right&amp;quot;&lt;br /&gt;
| __TOC__&lt;br /&gt;
|}&lt;br /&gt;
==Resource Summary View==&lt;br /&gt;
&lt;br /&gt;
To get started, users can query the overall status of resources on the cluster. The &#039;&#039;&#039;&amp;quot;qsum&amp;quot;&#039;&#039;&#039; script will list all queues and nodes, as well as how many are offline, down, free, or assigned to users. This is a script developed by our team, and may need to be updated if something goes wrong. Please contact us if you experience any problems.&lt;br /&gt;
&lt;br /&gt;
Each queue groups a number of nodes together based on their hardware and software configurations. Nodes can be part of more than one queue, and there are other complex details that we are ignoring here for the purpose of keeping it simple.&lt;br /&gt;
&lt;br /&gt;
===Queues===&lt;br /&gt;
&lt;br /&gt;
Here is a very brief summary of what each of the queues is, and how to use them efficiently:&lt;br /&gt;
&lt;br /&gt;
; a4000: This is a queue that has three machines with 16 cores each; each of these machines is furthermore equipped with three A4000 GPUs. That makes a total of 9 A4000 GPUs available to users. Neither the GPUs nor the processors are particularly powerful these days, but they make for a good software development environment. The machines have 512GB of memory, which makes them a good platform for experimenting with GPU capabilities.&lt;br /&gt;
; a6000: This is a queue that has only one single machine with 64 cores total, and is equipped with four A6000 GPUs. The system can be upgraded to 8 A6000 GPUs if needed. This is a decent GPU machine that can take a solid workload. The machine has 750GB of memory, which makes for a good production platform.&lt;br /&gt;
; amd16: This is a queue with many of our older AMD-based 16-core machines, each of which equipped with 32GB of memory. While individual machines are a bit outdated, they are all interconnected with Infiniband and can provide a solid production workload in multi-node jobs over MPI without blocking the more current (and thus expensive) systems.&lt;br /&gt;
; epyc1/epyc2: These are 2 separate queues with slightly different performance characteristics. Each of the groups is interconnected with Infiniband to provide a platform for large and demanding software packages, such as LS-Dyna and StarCCM+. They have between 256GB and 512GB of memory. Because licenses for these software packages (LS-Dyna and StarCCM+) are very expensive, these applications should use the two epyc queues for making optimum use of limited core licenses available to each package.&lt;br /&gt;
; xeon28: This is a set of intermediate machines with 28 cores and 64GB of memory. They can be used for a variety of purposes, including MPI jobs and single node application software.&lt;br /&gt;
; virtual: This is a set of nodes without MPI capabilities. They are virtual machines with 32GB each. They can be used for higher demand interactive applications that would interfere otherwise with other users on the login node machines. A user would submit interactive jobs to individual virtual machines and thus avoid any significant load on login nodes.&lt;br /&gt;
&lt;br /&gt;
===The Queue Summary Script (qsum)===&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
$ qsum&lt;br /&gt;
=============== a4000 ==========================================================&lt;br /&gt;
Queue: &amp;quot;a4000&amp;quot; / nodes: 3 / down: 0 / offline: 0 / busy: 0 / available: 3&lt;br /&gt;
      AVAILABLE (3): g001, g002, g003&lt;br /&gt;
=============== a6000 ==========================================================&lt;br /&gt;
Queue: &amp;quot;a6000&amp;quot; / nodes: 1 / down: 0 / offline: 0 / busy: 0 / available: 1&lt;br /&gt;
      AVAILABLE (1): lambda01&lt;br /&gt;
=============== amd16 ==========================================================&lt;br /&gt;
Queue: &amp;quot;amd16&amp;quot; / nodes: 33 / down: 2 / offline: 0 / busy: 2 / available: 29&lt;br /&gt;
           DOWN (2): n017, n030&lt;br /&gt;
            ley (2): n001, n002&lt;br /&gt;
     AVAILABLE (29): n003, n004, n005, n006, n007, n008, n009, n010, n011, n012&lt;br /&gt;
                     n013, n014, n015, n016, n018, n019, n020, n021, n022, n023&lt;br /&gt;
                     n024, n025, n026, n027, n028, n029, n031, n032, n039&lt;br /&gt;
=============== epyc1 ==========================================================&lt;br /&gt;
Queue: &amp;quot;epyc1&amp;quot; / nodes: 1 / down: 0 / offline: 0 / busy: 0 / available: 1&lt;br /&gt;
      AVAILABLE (1): a027&lt;br /&gt;
=============== epyc2 ==========================================================&lt;br /&gt;
Queue: &amp;quot;epyc2&amp;quot; / nodes: 20 / down: 0 / offline: 0 / busy: 5 / available: 15&lt;br /&gt;
            ley (2): a030, a031&lt;br /&gt;
         msitek (3): a028, a029, a032&lt;br /&gt;
     AVAILABLE (15): a033, a034, a035, a036, a037, a038, a039, a040, a041, a042&lt;br /&gt;
                     a043, a044, a045, a046, a047&lt;br /&gt;
=============== virtual ========================================================&lt;br /&gt;
Queue: &amp;quot;virtual&amp;quot; / nodes: 6 / down: 0 / offline: 0 / busy: 0 / available: 6&lt;br /&gt;
      AVAILABLE (6): v001, v002, v003, v004, v005, v006&lt;br /&gt;
=============== xeon28 =========================================================&lt;br /&gt;
Queue: &amp;quot;xeon28&amp;quot; / nodes: 12 / down: 0 / offline: 0 / busy: 0 / available: 12&lt;br /&gt;
     AVAILABLE (12): p001, p002, p003, p004, p005, p006, p007, p008, p009, p010&lt;br /&gt;
                     p011, p012&lt;br /&gt;
================================================================================&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
==Queue Status and Monitoring Jobs==&lt;br /&gt;
&lt;br /&gt;
===qstat===&lt;br /&gt;
&lt;br /&gt;
To find out out about the status of all running jobs on the cluster you can use the &#039;&#039;&#039;&amp;quot;qstat&amp;quot;&#039;&#039;&#039; command. Here is an example:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
$ qstat&lt;br /&gt;
&lt;br /&gt;
Nov 20 18:30 ley@login3:Plots$ qstat&lt;br /&gt;
Job id            Name             User              Time Use S Queue&lt;br /&gt;
----------------  ---------------- ----------------  -------- - -----&lt;br /&gt;
3023.pbs          STDIN            msitek            4144:14* R epyc2           &lt;br /&gt;
3029.pbs          STDIN            ley               76:46:53 R epyc2           &lt;br /&gt;
3032.pbs          STDIN            msitek            2879:52* R epyc2           &lt;br /&gt;
3033.pbs          STDIN            msitek            3687:29* R epyc2           &lt;br /&gt;
3048.pbs          foo.sh           james.cook               0 Q amd16           &lt;br /&gt;
3060.pbs          of13.sh          ley               310:47:* R epyc2           &lt;br /&gt;
3061.pbs          of13.sh          ley               308:37:* R epyc2           &lt;br /&gt;
3062.pbs          of13.sh          ley               308:02:* R epyc2           &lt;br /&gt;
3063.pbs          of13.sh          ley               308:15:* R epyc2&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
The first column shows the &#039;&#039;&#039;job id&#039;&#039;&#039;, a unique identifier for all jobs ever submitted to the cluster. This job id is important when killing jobs, or for other actions you may need to take.&lt;br /&gt;
&lt;br /&gt;
The next column shows the name of the job script. If the column shows &#039;&#039;&#039;STDIN&#039;&#039;&#039;, it means that this is an &#039;&#039;&#039;interactive job&#039;&#039;&#039; where a user can enter commands in a terminal window. This is particularly useful for model and software development task where the application has to be started and killed repeatedly.&lt;br /&gt;
&lt;br /&gt;
The owner of the job is shown next. These are the user names of the various people using the cluster.&lt;br /&gt;
&lt;br /&gt;
The last three columns indicate the current run time of the job, whether it is running (R) or waiting (Q) for execution. The last entry shows the queue in which the job is running.&lt;br /&gt;
&lt;br /&gt;
===qstat -an1===&lt;br /&gt;
&lt;br /&gt;
Adding a few options gives much more detail about each jobs.&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
qstat -an1&lt;br /&gt;
&lt;br /&gt;
Nov 20 13:09 ley@login3:Plots$ qstat -an1&lt;br /&gt;
&lt;br /&gt;
pbs: &lt;br /&gt;
                                                            Req&#039;d  Req&#039;d   Elap&lt;br /&gt;
Job ID          Username Queue    Jobname    SessID NDS TSK Memory Time  S Time&lt;br /&gt;
--------------- -------- -------- ---------- ------ --- --- ------ ----- - -----&lt;br /&gt;
3023.pbs        msitek   epyc2    STDIN      24360*   1  64  350gb 100:0 R 81:46 a028/0*64&lt;br /&gt;
3029.pbs        ley      epyc2    STDIN      21719*   2 128  100gb 200:0 R 72:31 a030/0*64+a031/0*64&lt;br /&gt;
3032.pbs        msitek   epyc2    STDIN      18102*   1  64  350gb 100:0 R 57:57 a029/0*64&lt;br /&gt;
3033.pbs        msitek   epyc2    STDIN      830486   1  64  350gb 100:0 R 57:53 a032/0*64&lt;br /&gt;
3048.pbs        james.c* amd16    foo.sh        --    1  28   30gb 01:00 Q   --   -- &lt;br /&gt;
3060.pbs        ley      epyc2    STDIN      763101   1  64  350gb 48:00 R 06:42 a033/0*64&lt;br /&gt;
3061.pbs        ley      epyc2    STDIN      763947   1  64  350gb 48:00 R 06:40 a034/0*64&lt;br /&gt;
3062.pbs        ley      epyc2    STDIN      761473   1  64  350gb 48:00 R 06:39 a035/0*64&lt;br /&gt;
3063.pbs        ley      epyc2    STDIN      766205   1  64  350gb 48:00 R 06:40 a036/0*64&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
In this table, you can see how many nodes and cores are being used by each job. For example, job 3029 of the user &amp;quot;ley&amp;quot; shows that it is running on 2 nodes using a total of 128 cores. In addition to the elapsed time, the table also show the reserved time for this job. This allow you to estimate when a job will be definitely finalized (or killed by the system if still running).&lt;br /&gt;
&lt;br /&gt;
The last column (without a header) is written because the option &#039;&#039;&#039;&amp;quot;-an1&amp;quot;&#039;&#039;&#039; was used. This useful to learn about which nodes are used by each job.&lt;br /&gt;
&lt;br /&gt;
===qstat -q===&lt;br /&gt;
&lt;br /&gt;
To learn more about the queues on the cluster, the &#039;&#039;&#039;&amp;quot;-q&amp;quot;&#039;&#039;&#039; option turns out to be useful.&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
$ qstat -q&lt;br /&gt;
&lt;br /&gt;
server: pbs&lt;br /&gt;
&lt;br /&gt;
Queue            Memory CPU Time Walltime Node   Run   Que   Lm  State&lt;br /&gt;
---------------- ------ -------- -------- ---- ----- ----- ----  -----&lt;br /&gt;
virtual            30gb    --       --       1     0     0   --   E R&lt;br /&gt;
a4000             500gb    --       --       1     0     0   --   E R&lt;br /&gt;
a6000             750gb    --       --       1     0     0   --   E R&lt;br /&gt;
xeon28             --      --       --       4     0     0   --   E R&lt;br /&gt;
amd16              --      --       --       8     0     1   --   E R&lt;br /&gt;
epyc2              --      --       --       2    14     0   --   E R&lt;br /&gt;
epyc1              --      --       --       2     0     0   --   E R&lt;br /&gt;
                                               ----- -----&lt;br /&gt;
                                                  14     1&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
For each queue, some basic values are displayed. The first three queues listed above have a default memory allocation as shown, and the column &#039;&#039;&#039;&amp;quot;Node&amp;quot;&#039;&#039;&#039; indicates the maximum number of nodes that can be asked for at job submission time. For example, you can request just one node for a job from the first three queues (because these are queues where MPI makes no sense). The &#039;&#039;&#039;&amp;quot;xeon28&amp;quot;&#039;&#039;&#039; queue also for a maximum of 4 nodes per MPI job. The &#039;&#039;&#039;&amp;quot;amd16&amp;quot;&#039;&#039;&#039; queue has a maximum of 8 nodes per job, and the &#039;&#039;&#039;&amp;quot;epyc1&amp;quot;&#039;&#039;&#039; and &#039;&#039;&#039;&amp;quot;epyc2&amp;quot;&#039;&#039;&#039; queues have maxima of two nodes per job. These limitations can be changed by the administrator as needed. As shown above, this will prevent inefficient resource requests.&lt;br /&gt;
&lt;br /&gt;
===qstat -f===&lt;br /&gt;
&lt;br /&gt;
The command &#039;&#039;&#039;&amp;quot;qstat -f -F json 3029&amp;quot;&#039;&#039;&#039; retrieves extremely detailed stats on the running job 3029. The result can be returned in JSON format to be ready for further processing (shown below).&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
$ qstat -f -F json 3029&lt;br /&gt;
{&lt;br /&gt;
    &amp;quot;timestamp&amp;quot;:1763705353,&lt;br /&gt;
    &amp;quot;pbs_version&amp;quot;:&amp;quot;23.06.06&amp;quot;,&lt;br /&gt;
    &amp;quot;pbs_server&amp;quot;:&amp;quot;pbs&amp;quot;,&lt;br /&gt;
    &amp;quot;Jobs&amp;quot;:{&lt;br /&gt;
        &amp;quot;3029.pbs&amp;quot;:{&lt;br /&gt;
            &amp;quot;Job_Name&amp;quot;:&amp;quot;STDIN&amp;quot;,&lt;br /&gt;
            &amp;quot;Job_Owner&amp;quot;:&amp;quot;ley@login4&amp;quot;,&lt;br /&gt;
            &amp;quot;resources_used&amp;quot;:{&lt;br /&gt;
                &amp;quot;cpupercent&amp;quot;:98,&lt;br /&gt;
                &amp;quot;cput&amp;quot;:&amp;quot;76:46:53&amp;quot;,&lt;br /&gt;
                &amp;quot;hpmem&amp;quot;:&amp;quot;0b&amp;quot;,&lt;br /&gt;
                &amp;quot;mem&amp;quot;:&amp;quot;52428800kb&amp;quot;,&lt;br /&gt;
                &amp;quot;ncpus&amp;quot;:128,&lt;br /&gt;
                &amp;quot;vmem&amp;quot;:&amp;quot;52428800kb&amp;quot;,&lt;br /&gt;
                &amp;quot;walltime&amp;quot;:&amp;quot;78:09:32&amp;quot;&lt;br /&gt;
            },&lt;br /&gt;
            &amp;quot;job_state&amp;quot;:&amp;quot;R&amp;quot;,&lt;br /&gt;
            &amp;quot;queue&amp;quot;:&amp;quot;epyc2&amp;quot;,&lt;br /&gt;
            &amp;quot;server&amp;quot;:&amp;quot;pbs&amp;quot;,&lt;br /&gt;
            &amp;quot;Checkpoint&amp;quot;:&amp;quot;u&amp;quot;,&lt;br /&gt;
            &amp;quot;ctime&amp;quot;:&amp;quot;Mon Nov 17 17:58:25 2025&amp;quot;,&lt;br /&gt;
            &amp;quot;Error_Path&amp;quot;:&amp;quot;/dev/pts/0&amp;quot;,&lt;br /&gt;
            &amp;quot;exec_host&amp;quot;:&amp;quot;a030/0*64+a031/0*64&amp;quot;,&lt;br /&gt;
            &amp;quot;exec_vnode&amp;quot;:&amp;quot;(a030:ncpus=64:mem=52428800kb)+(a031:ncpus=64:mem=52428800kb)&amp;quot;,&lt;br /&gt;
            &amp;quot;Hold_Types&amp;quot;:&amp;quot;n&amp;quot;,&lt;br /&gt;
            &amp;quot;interactive&amp;quot;:&amp;quot;True&amp;quot;,&lt;br /&gt;
            &amp;quot;Join_Path&amp;quot;:&amp;quot;n&amp;quot;,&lt;br /&gt;
            &amp;quot;Keep_Files&amp;quot;:&amp;quot;n&amp;quot;,&lt;br /&gt;
            &amp;quot;Mail_Points&amp;quot;:&amp;quot;a&amp;quot;,&lt;br /&gt;
            &amp;quot;mtime&amp;quot;:&amp;quot;Fri Nov 21 00:07:59 2025&amp;quot;,&lt;br /&gt;
            &amp;quot;Output_Path&amp;quot;:&amp;quot;/dev/pts/0&amp;quot;,&lt;br /&gt;
            &amp;quot;Priority&amp;quot;:0,&lt;br /&gt;
            &amp;quot;qtime&amp;quot;:&amp;quot;Mon Nov 17 17:58:25 2025&amp;quot;,&lt;br /&gt;
            &amp;quot;Rerunable&amp;quot;:&amp;quot;False&amp;quot;,&lt;br /&gt;
            &amp;quot;Resource_List&amp;quot;:{&lt;br /&gt;
                &amp;quot;mem&amp;quot;:&amp;quot;100gb&amp;quot;,&lt;br /&gt;
                &amp;quot;mpiprocs&amp;quot;:128,&lt;br /&gt;
                &amp;quot;ncpus&amp;quot;:128,&lt;br /&gt;
                &amp;quot;nodect&amp;quot;:2,&lt;br /&gt;
                &amp;quot;place&amp;quot;:&amp;quot;free&amp;quot;,&lt;br /&gt;
                &amp;quot;select&amp;quot;:&amp;quot;2:ncpus=64:mem=50gb:mpiprocs=64&amp;quot;,&lt;br /&gt;
                &amp;quot;walltime&amp;quot;:&amp;quot;200:00:00&amp;quot;&lt;br /&gt;
            },&lt;br /&gt;
            &amp;quot;stime&amp;quot;:&amp;quot;Mon Nov 17 17:58:25 2025&amp;quot;,&lt;br /&gt;
            &amp;quot;session_id&amp;quot;:2171964,&lt;br /&gt;
            &amp;quot;jobdir&amp;quot;:&amp;quot;/mnt/lustre/arrow/home/ley&amp;quot;,&lt;br /&gt;
            &amp;quot;substate&amp;quot;:42,&lt;br /&gt;
            &amp;quot;Variable_List&amp;quot;:{&lt;br /&gt;
                &amp;quot;PBS_O_HOME&amp;quot;:&amp;quot;/mnt/lustre/arrow/home/ley&amp;quot;,&lt;br /&gt;
                &amp;quot;PBS_O_LANG&amp;quot;:&amp;quot;en_US.UTF-8&amp;quot;,&lt;br /&gt;
                &amp;quot;PBS_O_LOGNAME&amp;quot;:&amp;quot;ley&amp;quot;,&lt;br /&gt;
                &amp;quot;PBS_O_PATH&amp;quot;:&amp;quot;/shared/apps/active/lstc/lsprepost/SP-4.5:/shared/apps/active/lstc/lsprepost/DP-4.3:/shared/bin:/usr/share/Modules/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/pbs/bin:/opt/thinlinc/bin:/opt/thinlinc/sbin:/mnt/lustre/arrow/home/ley/.local/bin:/mnt/lustre/arrow/home/ley/bin&amp;quot;,&lt;br /&gt;
                &amp;quot;PBS_O_MAIL&amp;quot;:&amp;quot;/var/spool/mail/ley&amp;quot;,&lt;br /&gt;
                &amp;quot;PBS_O_SHELL&amp;quot;:&amp;quot;/bin/bash&amp;quot;,&lt;br /&gt;
                &amp;quot;PBS_O_WORKDIR&amp;quot;:&amp;quot;/mnt/lustre/arrow/home/ley/Qualification/LS-Dyna/Rocky9/Seatbelt/Template&amp;quot;,&lt;br /&gt;
                &amp;quot;PBS_O_SYSTEM&amp;quot;:&amp;quot;Linux&amp;quot;,&lt;br /&gt;
                &amp;quot;PBS_O_QUEUE&amp;quot;:&amp;quot;epyc2&amp;quot;,&lt;br /&gt;
                &amp;quot;PBS_O_HOST&amp;quot;:&amp;quot;login4&amp;quot;&lt;br /&gt;
            },&lt;br /&gt;
            &amp;quot;comment&amp;quot;:&amp;quot;Job run at Mon Nov 17 at 17:58 on (a030:ncpus=64:mem=52428800kb)+(a031:ncpus=64:mem=52428800kb)&amp;quot;,&lt;br /&gt;
            &amp;quot;etime&amp;quot;:&amp;quot;Mon Nov 17 17:58:25 2025&amp;quot;,&lt;br /&gt;
            &amp;quot;run_count&amp;quot;:1,&lt;br /&gt;
            &amp;quot;Submit_arguments&amp;quot;:&amp;quot;-I -q epyc2 -l walltime=200:00:00,select=2:ncpus=64:mem=50gb:mpiprocs=64&amp;quot;,&lt;br /&gt;
            &amp;quot;project&amp;quot;:&amp;quot;_pbs_project_default&amp;quot;,&lt;br /&gt;
            &amp;quot;Submit_Host&amp;quot;:&amp;quot;login4&amp;quot;&lt;br /&gt;
        }&lt;br /&gt;
    }&lt;br /&gt;
}&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
===Manual pages for qstat===&lt;br /&gt;
&lt;br /&gt;
To learn more about the &#039;&#039;&#039;&amp;quot;qstat&amp;quot;&#039;&#039;&#039; command, you can use the command &#039;&#039;&#039;&amp;quot;man qstat&amp;quot;&#039;&#039;&#039;, which will print a lot of detailed information about the capabilities of this command.&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
$ man qstat&lt;br /&gt;
&lt;br /&gt;
qstat(1B)                                       PBS Professional                                       qstat(1B)&lt;br /&gt;
&lt;br /&gt;
NAME&lt;br /&gt;
       qstat - display status of PBS jobs, queues, or servers&lt;br /&gt;
&lt;br /&gt;
SYNOPSIS&lt;br /&gt;
       Displaying Job Status&lt;br /&gt;
              Default format:&lt;br /&gt;
              qstat [-E] [-J] [-p] [-t] [-w] [-x] [[&amp;lt;job ID&amp;gt; | &amp;lt;destination&amp;gt;] ...]&lt;br /&gt;
&lt;br /&gt;
              Long format:&lt;br /&gt;
              qstat -f [-F json | dsv [-D &amp;lt;delimiter&amp;gt;]] [-E] [-J] [-p] [-t] [-w]&lt;br /&gt;
                    [-x] [[&amp;lt;job ID&amp;gt; | &amp;lt;destination&amp;gt;] ...]&lt;br /&gt;
... &amp;lt;many more pages&amp;gt;&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
==Job Submission Basics==&lt;br /&gt;
&lt;br /&gt;
Jobs are submitted into the system using the &#039;&#039;&#039;&amp;quot;qsub&amp;quot;&#039;&#039;&#039; application. This application can take many different options and allows for a lot of different resource requests to tell the cluster what to do. We are running &#039;&#039;&#039;OpenPBS 23.06.06&#039;&#039;&#039; as our job scheduler. Here is a link to the User&#039;s Manual (of PBS PRO) if you want to explore gory details and capabilities. The User&#039;s Guide has about 240 pages, the Reference Guide has 500 pages, and the Big Book has 2500 pages. So there is a lot of information available. I also added job submission info for the LCRC cluster.&lt;br /&gt;
&lt;br /&gt;
* [https://argonne-lcrc.github.io/user-guides/running-jobs-at-lcrc/pbs-pro/ Argonne&#039;s LCRC pages on job submissions on their clusters]&lt;br /&gt;
* [https://help.altair.com/2022.1.0/PBS%20Professional/PBSUserGuide2022.1.pdf PBS Professional 2022.1 User&#039;s Guide]&lt;br /&gt;
* [https://help.altair.com/2022.1.0/PBS%20Professional/PBSReferenceGuide2022.1.pdf PBS Professional 2022.1 Reference Guide]&lt;br /&gt;
* [https://help.altair.com/2022.1.0/PBS%20Professional/PBS2022.1.pdf Altair PBS Professional 2022.1 Big Book]&lt;br /&gt;
&lt;br /&gt;
The User&#039;s Guide can be very helpful to clarify some of the concepts and capabilities, but it can be hard to find the specific information you may be looking for. Please understand that we are no longer running TORQUE and MAUI, so the syntax for job submission is distinctively different yet quite similar.&lt;br /&gt;
&lt;br /&gt;
The reference guide may be helpful to understand the complete syntax and full capabilities of the software.&lt;br /&gt;
&lt;br /&gt;
The big book is what I had to use when configuring OpenPBS earlier this year. This includes all the tricky details needed to make the system work smoothly for us. It&#039;s a bit scary to look at a PDF file that is 2500 pages long, but that is nothing compared to the StarCCM+ manuals.&lt;br /&gt;
&lt;br /&gt;
&amp;lt;BLOCKQUOTE&amp;gt;&lt;br /&gt;
[[File:Attention.jpg|25px]] &#039;&#039;&#039;IMPORTANT NOTE:&#039;&#039;&#039; &#039;&#039;The following sections are important to understand. They explain how jobs are submitted and then scjeduled for execution based on resources available and the specific need of the user.&#039;&#039;&lt;br /&gt;
&amp;lt;/BLOCKQUOTE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
The following sections explain the various tasks you may want to submit fir execution, ordered from simple to complex.&lt;br /&gt;
&lt;br /&gt;
* General Batch Jobs&lt;br /&gt;
** Requesting a Single Node for a Job&lt;br /&gt;
** Requesting Multiple Nodes for a Job&lt;br /&gt;
* Embedded Job Resource Requests&lt;br /&gt;
* Interactive Jobs&lt;br /&gt;
* Interactive Jobs with X-Windows GUI Applications&lt;br /&gt;
* Running Multiple Jobs on Single Nodes&lt;br /&gt;
* Running Jobs using GPUs&lt;br /&gt;
&lt;br /&gt;
===General Batch Jobs===&lt;br /&gt;
&lt;br /&gt;
Let&#039;s get started with a very basic usage of the system. Let&#039;s assume you have a simple application, and you want to execute it on a cluster node. Let&#039;s also assume that this is a very simple application, one that runs on one or a few cores, doesn&#039;t require any keyboard interaction with the user, doesn&#039;t need the user to see what&#039;s typically written to the screen, and writes its output to files. In this case, we can submit this application as a batch job, which will place it into an execution queue and process it as soon as a node becomes available.&lt;br /&gt;
&lt;br /&gt;
If the application requires more cores than a single node can provide, we can run the application over Infiniband with MPI message passing. In this case, we need to understand the concept of MPI applications a bit better. In both cases, we get started by creating a folder on the file system. Naming conventions are important so that you can distinguish the jobs by folder name.&lt;br /&gt;
&lt;br /&gt;
For both of the above scenarios, you would typically create a Bash shell script, and then submit the script into one of the queues for eventual execution.&lt;br /&gt;
&lt;br /&gt;
====Requesting a Single Node for a Job====&lt;br /&gt;
&lt;br /&gt;
Let&#039;s try something rather trivial to get used to the concept. Create yourself a folder, for example &#039;&#039;&#039;&amp;quot;myjobfolder&amp;quot;&#039;&#039;&#039;. Within that folder, create a job submission script. That script can have any name, but something short and simple may be best. Let&#039;s assume you create a file called &#039;&#039;&#039;&amp;quot;cluster.job&amp;quot;&#039;&#039;&#039;. The file doesn&#039;t have to have that extension. Any file name will do. But using the same filename for all of your jobs helps finding your way around the many files that will be created over time. The &#039;&#039;&#039;&amp;quot;cluster.job&amp;quot;&#039;&#039;&#039; file should look something like this:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
#!/bin/bash&lt;br /&gt;
#&lt;br /&gt;
# the following ensures that you will change into the directory where you are&lt;br /&gt;
# submitting the job from.&lt;br /&gt;
cd $PBS_O_WORKDIR&lt;br /&gt;
#&lt;br /&gt;
# now we sleep for 60 seconds and waste time. This is a placeholder for your application,&lt;br /&gt;
# which would be doing useful work for you.&lt;br /&gt;
sleep 60&lt;br /&gt;
#&lt;br /&gt;
# and after doing things, we may want to write something into a file to show that&lt;br /&gt;
# our jobs is done.&lt;br /&gt;
echo `date` &amp;gt; info.log&lt;br /&gt;
#&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
This can be submitted without detailed resource specifications (except for the walltime, which is be default 0:00:00):&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
$ qsub -q virtual -l walltime=1:00:00 cluster.job &lt;br /&gt;
3072.pbs&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
Wait a little, then check the status of running jobs:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
$ qstat -an1&lt;br /&gt;
&lt;br /&gt;
pbs: &lt;br /&gt;
                                                            Req&#039;d  Req&#039;d   Elap&lt;br /&gt;
Job ID          Username Queue    Jobname    SessID NDS TSK Memory Time  S Time&lt;br /&gt;
--------------- -------- -------- ---------- ------ --- --- ------ ----- - -----&lt;br /&gt;
3023.pbs        msitek   epyc2    STDIN      24360*   1  64  350gb 100:0 R 83:17 a028/0*64&lt;br /&gt;
3029.pbs        ley      epyc2    STDIN      21719*   2 128  100gb 200:0 R 74:00 a030/0*64+a031/0*64&lt;br /&gt;
3033.pbs        msitek   epyc2    STDIN      830486   1  64  350gb 100:0 R 59:23 a032/0*64&lt;br /&gt;
3048.pbs        james.c* amd16    foo.sh        --    1  28   30gb 01:00 Q   --   -- &lt;br /&gt;
3060.pbs        ley      epyc2    STDIN      763101   1  64  350gb 48:00 R 08:10 a033/0*64&lt;br /&gt;
3061.pbs        ley      epyc2    STDIN      763947   1  64  350gb 48:00 R 08:10 a034/0*64&lt;br /&gt;
3070.pbs        ley      epyc2    STDIN      766847   1  64  350gb 48:00 R 07:23 a042/0*64&lt;br /&gt;
3072.pbs        ley      virtual  cluster.j* 230230   1   4   30gb 01:00 E 00:01 v001/0*4&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
In this particular example, we are sending this job to the &#039;&#039;&#039;queue &amp;quot;virtual&amp;quot;&#039;&#039;&#039;. This queue, by default, allocates 30GB of memory to the job, and runs on 1 node with 4 cores. This is sufficient capacity to run quite a workload. When submitting a job to a single node, &#039;&#039;&#039;reasonable maximum allocations are automatically assigned&#039;&#039;&#039;, and the user doesn&#039;t have to worry about running out of memory or how many cores he will be using.&lt;br /&gt;
&lt;br /&gt;
The only required argument is the &#039;&#039;&#039;&amp;quot;walltime&amp;quot;&#039;&#039;&#039; argument. By default, the job will quit as soon as it is submitted. This indicates to the user that he forgot to provide the &#039;&#039;&#039;&amp;quot;walltime&amp;quot;&#039;&#039;&#039; argument.&lt;br /&gt;
&lt;br /&gt;
When the job disappears from the job list, it is done. At this point, you will find the file &amp;quot;info.log&amp;quot; in your job folder.&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
$ cat info.log &lt;br /&gt;
Thu Nov 20 08:00:31 PM CST 2025&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
====Requesting Multiple Nodes for a Job====&lt;br /&gt;
&lt;br /&gt;
To run jobs on multiple nodes, you will be likely &#039;&#039;&#039;executing jobs using MPI&#039;&#039;&#039;, the message passing interface. This establishes high-speed low-latency interconnections between the cores on one machine and the cores on the other machines. Data transfer does not require involvement of the cores themselves. Instead, the core tell the InfiniBand interconnect (and cores on the same node through shared memory) to transfer the data through RDMA, remoted direct memory access. The cores don&#039;t need to spend CPU cycles on copying data, but rather simply access the data once it has been copied by the Infiniband fabric. This makes for extremely efficient remote memory access, and message passing is used to coordinate data transfer between the cores no matter where they are located on any of the nodes.&lt;br /&gt;
&lt;br /&gt;
On our cluster, MPI-aware applications like &#039;&#039;&#039;OpenFOAM&#039;&#039;&#039;, &#039;&#039;&#039;StarCCM+&#039;&#039;&#039;, and &#039;&#039;&#039;LS-Dyna&#039;&#039;&#039; can be loaded as modules, which then automatically selects the most appropriate MPI library to use. The software applications have been tested to ensure that they work out-of-the box if a user selects any specific version of any of the applications.&lt;br /&gt;
&lt;br /&gt;
The following is a very trivial example for the MPI execution of a very simple executable, with one copy running on each core of the nodes allocated to  the job. It doesn&#039;t perform any real work and just wastes resources for a short time, but it illustrates how execution on the cores of various nodes works.&lt;br /&gt;
&lt;br /&gt;
Like in the previous section, we start with a simple job script that we submit to an appropriate queues. In this case, we pick a queue that has machines with Infiniband interfaces supporting efficient communications. Let&#039;s assume we edit a file with the name &#039;&#039;&#039;&amp;quot;parallel.job&amp;quot;&#039;&#039;&#039; like this:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
#!/bin/bash&lt;br /&gt;
#&lt;br /&gt;
# the following ensures that you will change into the directory where you are&lt;br /&gt;
# submitting the job from.&lt;br /&gt;
cd $PBS_O_WORKDIR&lt;br /&gt;
#&lt;br /&gt;
# to execute a simple command on all of the cores of all of the nodes allocated to the job,&lt;br /&gt;
# we need to make one of the MPI versions available. Let&#039;s use one of the most up-to-date&lt;br /&gt;
# MPI library available on the cluster&lt;br /&gt;
module load intel/2024.2.0/mpi/2021.13&lt;br /&gt;
#&lt;br /&gt;
# now we are apply a few settings that ensure that the MPI library will use the highest-performing&lt;br /&gt;
# Infiniband Interconnect, as well as a few options to tell MPI how to interface nodes with&lt;br /&gt;
# each other and which specific Infiniband adapter to use. This is complex and requires in-depth&lt;br /&gt;
# knowledge of the QLogic Infiniband adapters we are using. It is unlikely that you will ever have to&lt;br /&gt;
# deal with these options, because the &amp;quot;module load&amp;quot; command for the engineering applications we provide&lt;br /&gt;
# on ARROW will handle all those details transparently without the user needing to understand the details.&lt;br /&gt;
export I_MPI_HYDRA_BOOTSTRAP=ssh&lt;br /&gt;
export MPI_DEVICE=rdma:ofa-v2-ib0&lt;br /&gt;
export UCX_NET_DEVICES=qib0:1&lt;br /&gt;
#&lt;br /&gt;
# it doesn&#039;t make much sense, but in this example we are executing the OS command &amp;quot;uptime&amp;quot; on all cores&lt;br /&gt;
# of the nodes allocated to this job. The output from each core is written to the file info.log. We&lt;br /&gt;
# will find 56 lines of output in the file info.log, each created by the corresponding core executing&lt;br /&gt;
# the uptime command.&lt;br /&gt;
mpirun uptime &amp;gt; info.log&lt;br /&gt;
#&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
A good queue to test scripts is the &#039;&#039;&#039;&amp;quot;xeon28&amp;quot;&#039;&#039;&#039; queue. In the queue, we have 2 14-core Xeon processers per node, so that means that each node has 56 actual cores. We do not consider hyperthreading when doing parallel computing. 56 actual cores is what&#039;s being used here. The job submission will look like this:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
qsub -q xeon28 -l walltime=1:0:0 -l select=2:ncpus=28:mpiprocs=28:mem=60G parallel.job&lt;br /&gt;
        ^                  ^               ^       ^           ^      ^ ^ ^&lt;br /&gt;
        |                  |               |       |           |      | | + --- the name of the job script to execute&lt;br /&gt;
        |                  |               |       |           |      | + ----- don&#039;t forget to specify gigabytes&lt;br /&gt;
        |                  |               |       |           |      + ------- the amount of memory to request per node&lt;br /&gt;
        |                  |               |       |           + -------------- the number of MPI tasks per nodes&lt;br /&gt;
        |                  |               |       + -------------------------- the number of cores per node&lt;br /&gt;
        |                  |               + ---------------------------------- the number of nodes to select in the queue&lt;br /&gt;
        |                   + ------------------------------------------------- the requested time, in this case 1h&lt;br /&gt;
        + --------------------------------------------------------------------- the queue to be used for the job&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
At this point, the job has created a file &amp;quot;info.log&amp;quot; with 56 lines, one per core:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
 22:26:05 up 23 days,  9:28,  0 users,  load average: 0.00, 0.00, 0.00&lt;br /&gt;
 22:26:05 up 23 days,  9:28,  0 users,  load average: 0.00, 0.00, 0.00&lt;br /&gt;
 22:26:05 up 23 days,  9:28,  0 users,  load average: 0.00, 0.00, 0.00&lt;br /&gt;
 22:26:05 up 23 days,  9:28,  0 users,  load average: 0.00, 0.00, 0.00&lt;br /&gt;
 22:26:05 up 23 days,  9:28,  0 users,  load average: 0.00, 0.00, 0.00&lt;br /&gt;
 22:26:05 up 23 days,  9:28,  0 users,  load average: 0.00, 0.00, 0.00&lt;br /&gt;
 22:26:05 up 23 days,  9:53,  0 users,  load average: 0.06, 0.03, 0.00&lt;br /&gt;
 22:26:05 up 23 days,  9:53,  0 users,  load average: 0.06, 0.03, 0.00&lt;br /&gt;
 22:26:05 up 23 days,  9:53,  0 users,  load average: 0.06, 0.03, 0.00&lt;br /&gt;
...&lt;br /&gt;
&amp;lt;/PRE&amp;gt; &lt;br /&gt;
&lt;br /&gt;
In this simple example, the lines look all the same. Upon close examination through, you can find slightly different values for some of the lines. Some lines say that the machine is up for 23 days and 9:28, while others say 23 days and 9:53. Because all 28 cores of a node would see the same uptime of the server, half of the entries show one time stamp, and the other 28 cores show the other one. That demonstrates that the 56 processes have been running independently on 2 nodes.&lt;br /&gt;
&lt;br /&gt;
===Embedded Job Resource Requests===&lt;br /&gt;
&lt;br /&gt;
The job script can be modified to embed the resource requests in form of a series of &#039;&#039;&#039;#PBS&#039;&#039;&#039; statements at the beginning of the script file. This is a very common practice use at many HPC installations and job submission engines. Let&#039;s go back to the previous example where we run the script on two nodes in parallel. That is the &#039;&#039;&#039;&amp;quot;parallel.job&amp;quot;&#039;&#039;&#039; script file again:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
#!/bin/bash&lt;br /&gt;
#&lt;br /&gt;
#PBS -q xeon28&lt;br /&gt;
#PBS -l walltime=1:0:0&lt;br /&gt;
#PBS -l select=2:ncpus=28:mpiprocs=28:mem=60G&lt;br /&gt;
#&lt;br /&gt;
# the following ensures that you will change into the directory where you are&lt;br /&gt;
# submitting the job from.&lt;br /&gt;
cd $PBS_O_WORKDIR&lt;br /&gt;
#&lt;br /&gt;
# to execute a simple command on all of the cores of all of the nodes allocated to the job,&lt;br /&gt;
# we need to make one of the MPI versions available. Let&#039;s use one of the most up-to-date&lt;br /&gt;
# MPI library available on the cluster&lt;br /&gt;
module load intel/2024.2.0/mpi/2021.13&lt;br /&gt;
#&lt;br /&gt;
# now we are apply a few settings that ensure that the MPI library will use the highest-performing&lt;br /&gt;
# Infiniband Interconnect, as well as a few options to tell MPI how to interface nodes with&lt;br /&gt;
# each other and which specific Infiniband adapter to use. This is complex and requires in-depth&lt;br /&gt;
# knowledge of the QLogic Infiniband adapters we are using. It is unlikely that you will ever have to&lt;br /&gt;
# deal with these options, because the &amp;quot;module load&amp;quot; command for the engineering applications we provide&lt;br /&gt;
# on ARROW will handle all those details transparently without the user needing to understand the details.&lt;br /&gt;
export I_MPI_HYDRA_BOOTSTRAP=ssh&lt;br /&gt;
export MPI_DEVICE=rdma:ofa-v2-ib0&lt;br /&gt;
export UCX_NET_DEVICES=qib0:1&lt;br /&gt;
#&lt;br /&gt;
# it doesn&#039;t make much sense, but in this example we are executing the OS command &amp;quot;uptime&amp;quot; on all cores&lt;br /&gt;
# of the nodes allocated to this job. The output from each core is written to the file info.log. We&lt;br /&gt;
# will find 56 lines of output in the file info.log, each created by the corresponding core executing&lt;br /&gt;
# the uptime command.&lt;br /&gt;
mpirun uptime &amp;gt; info.log&lt;br /&gt;
#&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
If the resource requests are embedded within the file, they don&#039;t have to be specified on the command line any longer (the command line overrides the embedded specifications though). This may be convenient, because all the user has to do for job submission is the following:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
qsub parallel.job&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
Here is an example with more resource specifications and job settings that affect the behavior of the job:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
#!/bin/bash&lt;br /&gt;
#&lt;br /&gt;
#PBS -q xeon28&lt;br /&gt;
#PBS -l walltime=1:0:0&lt;br /&gt;
#PBS -l select=2:ncpus=28:mpiprocs=28:mem=60G&lt;br /&gt;
#PBS -A Account&lt;br /&gt;
#PBS -j oe&lt;br /&gt;
#PBS -N JobName&lt;br /&gt;
#PBS -e log.error&lt;br /&gt;
#PBS -o log.output&lt;br /&gt;
#PBS -m bae&lt;br /&gt;
#&lt;br /&gt;
...&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
I leave this to you as an exercise to figure out what the various options mean and how to specify them. There are many more, all documented in the PBS PRO manual (see above). Most of them are not terribly relevant and can be omitted.&lt;br /&gt;
&lt;br /&gt;
===Interactive Jobs===&lt;br /&gt;
&lt;br /&gt;
On ARROW, we don&#039;t restrict queues to be used only in batch mode. While &#039;&#039;&#039;batch mode&#039;&#039;&#039; is efficient for lining up a lot of work to be executed one after the other, ARROW has been designed to &#039;&#039;&#039;allow efficient model and software development in interactive mode&#039;&#039;&#039;. We have always ensured to have more computers than minimally needed to make it possible to dedicate resources to developers as needed, even if that means wasted CPU cycles. At times, we may ask you to limit the number of interactive jobs so that a large batch workload can be processed efficiently. This happens from time to time, and we have our users coordinate this with each other.&lt;br /&gt;
&lt;br /&gt;
Let&#039;s assume that you are developing an MPI application, or you are working on a complex &#039;&#039;&#039;OpenFOAM&#039;&#039;&#039; model that requires to start parallel processes over and over again just to find a bug and then fix it quickly. To do that, you can &#039;&#039;&#039;request an interactive job&#039;&#039;&#039; by adding the &#039;&#039;&#039;&amp;quot;-I&amp;quot;&#039;&#039;&#039; option to the job submission command (this is an uppercase I). Let&#039;s go to the parallel multi-node example from above:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
qsub -I -q xeon28 -l walltime=1:0:0 -l select=2:ncpus=28:mpiprocs=28:mem=60G&lt;br /&gt;
      ^    ^                  ^               ^       ^           ^      ^ ^&lt;br /&gt;
      |    |                  |               |       |           |      | + --- don&#039;t forget to specify gigabytes&lt;br /&gt;
      |    |                  |               |       |           |      + ----- the amount of memory to request per node&lt;br /&gt;
      |    |                  |               |       |           + ------------ the number of MPI tasks per nodes&lt;br /&gt;
      |    |                  |               |       + ------------------------ the number of cores per node&lt;br /&gt;
      |    |                  |               + -------------------------------- the number of nodes to select in the queue&lt;br /&gt;
      |    |                   + ----------------------------------------------- the requested time, in this case 1h&lt;br /&gt;
      |    + ------------------------------------------------------------------- the queue to be used for the job&lt;br /&gt;
      + ------------------------------------------------------------------------ request an interactive job &amp;lt;&amp;lt;===&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
When running interactive jobs with the &#039;&#039;&#039;&amp;quot;-I&amp;quot;&#039;&#039;&#039; parameter, we don&#039;t specify av job script at the end of the submission command. The interactive job will instead start (once the nodes are available) in interactive mode, meaning that the terminal session changes over from being a series of commands executed on the login server to being a series of commands being executed on the first node of the group of nodes that are allocated to the job. At this point, you can change to the desired working directory, but what you do with the allocated resources is entirely up to you. You can load modules, including MPI libraries, and then issue the commands for your application interactively and see how they execute. If you start an &#039;&#039;&#039;&amp;quot;mpirun&amp;quot;&#039;&#039;&#039;, the cores on your allocated secondary node will work as expected. There is no difference to batch mode, other than you having the ability to execute lines of commands at will.&lt;br /&gt;
&lt;br /&gt;
===Interactive Jobs with X-Windows GUI Applications===&lt;br /&gt;
&lt;br /&gt;
Interactive use can go further than that. With some of our software applications, like &#039;&#039;&#039;StarCCM+&#039;&#039;&#039;, you can run an &#039;&#039;&#039;interactive GUI application&#039;&#039;&#039; where you control the computational work from within the applications&#039; GUI. Within the GUI, you can control execution of the numerical solver that runs on as many cores as you requested, while being able to reconfigure the case through the GUI as well. Furthermore, you can visualize developing results on the fly by creating complex plots and visualizations.&lt;br /&gt;
&lt;br /&gt;
All that is need is an option &#039;&#039;&#039;&amp;quot;-X&amp;quot;&#039;&#039;&#039; being used as part of the job submission, like this:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
qsub -X -I -q xeon28 -l walltime=1:0:0 -l select=2:ncpus=28:mpiprocs=28:mem=60G&lt;br /&gt;
      ^  ^    ^                  ^               ^       ^           ^      ^ ^&lt;br /&gt;
      |  |    |                  |               |       |           |      | + --- don&#039;t forget to specify gigabytes&lt;br /&gt;
      |  |    |                  |               |       |           |      + ----- the amount of memory to request per node&lt;br /&gt;
      |  |    |                  |               |       |           + ------------ the number of MPI tasks per nodes&lt;br /&gt;
      |  |    |                  |               |       + ------------------------ the number of cores per node&lt;br /&gt;
      |  |    |                  |               + -------------------------------- the number of nodes to select in the queue&lt;br /&gt;
      |  |    |                   + ----------------------------------------------- the requested time, in this case 1h&lt;br /&gt;
      |  |    + ------------------------------------------------------------------- the queue to be used for the job&lt;br /&gt;
      |  + ------------------------------------------------------------------------ request an interactive job&lt;br /&gt;
      + --------------------------------------------------------------------------- request GUI capabilities &amp;lt;&amp;lt;===&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
===Running Multiple Jobs on Single Nodes===&lt;br /&gt;
&lt;br /&gt;
A feature that is new on ARROW is the ability to run multiple jobs on a single node. Let&#039;s assume that you are performing a sensitivity analysis on an existing model, and the model is simple enough to return results within a reasonable time on just a few cores of a higher end machine (maybe you are running SMP versions of &#039;&#039;&#039;LS-Dyna&#039;&#039;&#039;). Our high end machines have 64 cores, so lets assume you have an &#039;&#039;&#039;LS-Dyna&#039;&#039;&#039; model that runs well on 8 cores and doesn&#039;t use a whole lot of memory. In this case, you can submit individual jobs that request simply 8 cores and a fraction of the available memory available on the node, and all jobs execute independently from each other. Each job is fit into a slot where available. It is not very different from using whole nodes for everything. The important consideration is that each job is cleanly constrained into it&#039;s allotted resources using the &#039;&#039;&#039;CGROUPS&#039;&#039;&#039; functionality of modern operating systems. Because an abusive user cannot use more cores or more memory than allocated to his job, other users can safely run smaller jobs on the same node.&lt;br /&gt;
&lt;br /&gt;
Lets assume that we have a number of smaller jobs that we want to run on a single node in the &#039;&#039;&#039;&amp;quot;xeon28&amp;quot;&#039;&#039;&#039; queue. Each job would be submitted by using reduced resources that allow for sharing but that  guarantee that the jobs will be run successfully. In this case, you can &#039;&#039;&#039;submit many jobs&#039;&#039;&#039; in the following manner (with a job script for the small jobs, each of which can &#039;&#039;&#039;request varying resources&#039;&#039;&#039; if needed - some may want to run on 5 cores, others on 3):&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
#!/bin/bash&lt;br /&gt;
#&lt;br /&gt;
# the following ensures that you will change into the directory where you are&lt;br /&gt;
# submitting the job from.&lt;br /&gt;
cd $PBS_O_WORKDIR&lt;br /&gt;
#&lt;br /&gt;
# now we sleep for 300 seconds and waste time. This is a placeholder for your application,&lt;br /&gt;
# which would be doing useful work for you.&lt;br /&gt;
sleep 300&lt;br /&gt;
#&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
Now we submit a variety of these jobs (11 total in this example) to the &#039;&#039;&#039;&amp;quot;xeon28&amp;quot;&#039;&#039;&#039; queue for execution (note that the first few jobs request different amounts of memory and core counts):&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
qsub -q xeon28 -l walltime=1:0:0 -l select=1:ncpus=12:mpiprocs=12:mem=5G small.job &lt;br /&gt;
qsub -q xeon28 -l walltime=1:0:0 -l select=1:ncpus=10:mpiprocs=10:mem=7G small.job &lt;br /&gt;
qsub -q xeon28 -l walltime=1:0:0 -l select=1:ncpus=8:mpiprocs=8:mem=9G small.job &lt;br /&gt;
qsub -q xeon28 -l walltime=1:0:0 -l select=1:ncpus=16:mpiprocs=16:mem=20G small.job &lt;br /&gt;
qsub -q xeon28 -l walltime=1:0:0 -l select=1:ncpus=2:mpiprocs=2:mem=2G small.job &lt;br /&gt;
qsub -q xeon28 -l walltime=1:0:0 -l select=1:ncpus=2:mpiprocs=2:mem=2G small.job &lt;br /&gt;
qsub -q xeon28 -l walltime=1:0:0 -l select=1:ncpus=2:mpiprocs=2:mem=2G small.job &lt;br /&gt;
qsub -q xeon28 -l walltime=1:0:0 -l select=1:ncpus=2:mpiprocs=2:mem=2G small.job &lt;br /&gt;
qsub -q xeon28 -l walltime=1:0:0 -l select=1:ncpus=2:mpiprocs=2:mem=2G small.job &lt;br /&gt;
qsub -q xeon28 -l walltime=1:0:0 -l select=1:ncpus=2:mpiprocs=2:mem=2G small.job &lt;br /&gt;
qsub -q xeon28 -l walltime=1:0:0 -l select=1:ncpus=2:mpiprocs=2:mem=2G small.job &lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
They are now running in the order of submission, allocated on as few nodes in the &amp;quot;xeon28&amp;quot; queue as necessary. Only 2 nodes are being loaded quite heavily, and 4 more cores are in use on a third node.&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
Nov 20 23:34 ley@login3:myjobfolder$ qstat -an1&lt;br /&gt;
&lt;br /&gt;
pbs: &lt;br /&gt;
                                                            Req&#039;d  Req&#039;d   Elap&lt;br /&gt;
Job ID          Username Queue    Jobname    SessID NDS TSK Memory Time  S Time&lt;br /&gt;
--------------- -------- -------- ---------- ------ --- --- ------ ----- - -----&lt;br /&gt;
&lt;br /&gt;
3082.pbs        ley      xeon28   small.job  813221   1  12    5gb 01:00 R 00:01 p001/0*12&lt;br /&gt;
3083.pbs        ley      xeon28   small.job  813288   1  10    7gb 01:00 R 00:01 p001/1*10&lt;br /&gt;
3084.pbs        ley      xeon28   small.job  671792   1   8    9gb 01:00 R 00:01 p002/0*8&lt;br /&gt;
3085.pbs        ley      xeon28   small.job  671845   1  16   20gb 01:00 R 00:01 p002/1*16&lt;br /&gt;
3086.pbs        ley      xeon28   small.job  813361   1   2    2gb 01:00 R 00:00 p001/2*2&lt;br /&gt;
3087.pbs        ley      xeon28   small.job  813413   1   2    2gb 01:00 R 00:00 p001/3*2&lt;br /&gt;
3088.pbs        ley      xeon28   small.job  813464   1   2    2gb 01:00 R 00:00 p001/4*2&lt;br /&gt;
3089.pbs        ley      xeon28   small.job  671912   1   2    2gb 01:00 R 00:00 p002/2*2&lt;br /&gt;
3090.pbs        ley      xeon28   small.job  671969   1   2    2gb 01:00 R 00:00 p002/3*2&lt;br /&gt;
3091.pbs        ley      xeon28   small.job  632092   1   2    2gb 01:00 R 00:00 p003/0*2&lt;br /&gt;
3092.pbs        ley      xeon28   small.job  632100   1   2    2gb 01:00 R 00:00 p003/1*2&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
This is a particularly effective strategy to run concurrently many cases that don&#039;t scale well beyond a few cores. When running them on fewer cores but many of them at the same time, the overall processing rate will be much higher than executing them the traditional way.&lt;br /&gt;
&lt;br /&gt;
===Running Jobs using GPUs===&lt;br /&gt;
&lt;br /&gt;
The principle of running multiple jobs on a single node becomes particularly important when using servers equipped with &#039;&#039;&#039;GPUs&#039;&#039;&#039; for &#039;&#039;&#039;ML/AI&#039;&#039;&#039; applications. The cluster doesn&#039;t have a whole lot of &#039;&#039;&#039;GPUs&#039;&#039;&#039; at this point. We have three machines with three &#039;&#039;&#039;A4000&#039;&#039;&#039; GOUs, a &#039;&#039;&#039;total of 9 A4000 GPUs&#039;&#039;&#039;. Then we have a much more powerful single machine with our &#039;&#039;&#039;four A6000 GPUs&#039;&#039;&#039;.&lt;br /&gt;
&lt;br /&gt;
Using multiple GPUs in a single application is still something where the software has to be designed with hardware in mind. GPUs have several methods of communicating with each other, e.g. very fast &#039;&#039;&#039;NVLINK&#039;&#039;&#039; between pairs of GPUs, GPUs being directly connected to one of the two CPUs in the system and thus being able to communicate faster, and GPUs that have to jump between processors when communicating, and then the whole issue of having to go possibly through PCIe bridges.&lt;br /&gt;
&lt;br /&gt;
On our system, we are providing the ability to &#039;&#039;&#039;work mostly with individual GPUs&#039;&#039;&#039;. Users can also reserve entire nodes and develop or run applications that are adapted to that hardware, including several GPUs installed on that node. One thing we do not provide is the ability of GPU to GPU communication between nodes. Thus, a job cannot request more than one GPU node at a time.&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
qsub -q a4000 -I -l walltime=1:0:0 -l select=1:ncpus=5:mem=150G:ngpus=1&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
With these specifications, three single GPU jobs can run on a single server. Each job sees only one of the reserved GPU.&lt;br /&gt;
&lt;br /&gt;
To run a massive GPU job on 64 cores with 4 &#039;&#039;&#039;A6000 GPUs&#039;&#039;&#039;, submit the job like this:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
qsub -q a6000 -I -l walltime=1:0:0 -l select=1:ncpus=64:mem=725G:ngpus=4&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
===Manual pages for qsub===&lt;br /&gt;
&lt;br /&gt;
To learn more about the &amp;quot;qsub&amp;quot; command, you can use the command &amp;quot;man qsub&amp;quot;, which will print a lot of detailed information about the capabilities of this command.&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
$ man qsub&lt;br /&gt;
&lt;br /&gt;
qsub(1B)                                               PBS Professional                                              qsub(1B)&lt;br /&gt;
&lt;br /&gt;
NAME&lt;br /&gt;
       qsub - submit a job to PBS&lt;br /&gt;
&lt;br /&gt;
SYNOPSIS&lt;br /&gt;
       qsub [-a &amp;lt;date and time&amp;gt;] [-A &amp;lt;account string&amp;gt;] [-c &amp;lt;checkpoint spec&amp;gt;]&lt;br /&gt;
            [-C &amp;lt;directive prefix&amp;gt;] [-e &amp;lt;path&amp;gt;] [-f] [-h]&lt;br /&gt;
            [-I [-G [-- &amp;lt;GUI application/script&amp;gt;]] | [-X]] [-j &amp;lt;join&amp;gt;]&lt;br /&gt;
            [-J &amp;lt;range&amp;gt; [%&amp;lt;max subjobs]] [-k &amp;lt;discard&amp;gt;] [-l &amp;lt;resource list&amp;gt;]&lt;br /&gt;
            [-m &amp;lt;mail events&amp;gt;] [-M &amp;lt;user list&amp;gt;] [-N &amp;lt;name&amp;gt;] [-o &amp;lt;path&amp;gt;]&lt;br /&gt;
            [-p &amp;lt;priority&amp;gt;] [-P &amp;lt;project&amp;gt;] [-q &amp;lt;destination&amp;gt;] [-r &amp;lt;y|n&amp;gt;]&lt;br /&gt;
            [-R &amp;lt;remove options&amp;gt;] [-S &amp;lt;path list&amp;gt;] [-u &amp;lt;user list&amp;gt;]&lt;br /&gt;
            [-v &amp;lt;variable list&amp;gt;] [-V] [-W &amp;lt;additional attributes&amp;gt;] [-z]&lt;br /&gt;
            [- | &amp;lt;script&amp;gt; | -- &amp;lt;executable&amp;gt; [&amp;lt;arguments to executable&amp;gt;]]&lt;br /&gt;
       qsub --version&lt;br /&gt;
&lt;br /&gt;
DESCRIPTION&lt;br /&gt;
       You use the qsub command to submit a batch job to PBS.  Submitting a PBS job specifies a task, requests resources, and&lt;br /&gt;
       sets job attributes.&lt;br /&gt;
... &amp;lt;many more pages&amp;gt;&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
==LS-Dyna on the ARROW Cluster==&lt;br /&gt;
&lt;br /&gt;
===Currently Available LS-Dyna Versions===&lt;br /&gt;
&lt;br /&gt;
The following is a list of &#039;&#039;&#039;LS-Dyna versions&#039;&#039;&#039; available on &#039;&#039;&#039;ARROW&#039;&#039;&#039; after the latest reconfiguration of the system. As per LSTC/ANSYS, &#039;&#039;&#039;versions before 14.0.0 are not necessarily fully supported any longer&#039;&#039;&#039; because they are supposedly not compatible with modern operating systems and cannot be made to work reliably. We have tested the listed older versions of LS-Dyna and they have passed basic tests. They may not behave exactly as they did on the old CentOS 7 operating system, and time will show whether they can still be used or whether you will need to update your models and use a fully supported version.&lt;br /&gt;
&lt;br /&gt;
All versions are loaded using the &#039;&#039;&#039;&amp;quot;module load&amp;quot;&#039;&#039;&#039; command. Versions can be listed with the &#039;&#039;&#039;&amp;quot;module avail ls-dyna&amp;quot;&#039;&#039;&#039; command. To load one of the modules, use the following syntax:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
module load ls-dyna/14.2.0/mpi-d8-ifort190-avx512&lt;br /&gt;
            ^       ^      ^   ^  ^        ^&lt;br /&gt;
            |       |      |   |  |        + --- specify the extended instruction set needed for execution&lt;br /&gt;
            |       |      |   |  + ------------ load the version of the compiler that was used to create this &lt;br /&gt;
            |       |      |   + --------------- load the version that supports double precision variables&lt;br /&gt;
            |       |      + ------------------- load the MPP (MPI) version of LS-Dyna&lt;br /&gt;
            |       + -------------------------- load specifically version 14.2.0&lt;br /&gt;
            + ---------------------------------- load a version of LS-Dyna&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
The version string is composed of multiple elements to indicate variants in compilers and compiler options. Use the following guideline to choose an appropriate version to load:&lt;br /&gt;
&lt;br /&gt;
* &#039;&#039;&#039;&amp;quot;1&amp;quot;&#039;&#039;&#039; or &#039;&#039;&#039;&amp;quot;mpi&amp;quot;&#039;&#039;&#039; indicates whether this is a single node version of LS-Dyna (&#039;&#039;&#039;SMP&#039;&#039;&#039;) or whether this is a multi-node MPI version (&#039;&#039;&#039;MPP&#039;&#039;&#039;). All MPI versions use the &#039;&#039;&#039;IntelMPI 2022&#039;&#039;&#039; libraries which have been tested thoroughly on ARROW. MPI versions will use the Infiniband Network of ARROW for high-speed and low-latency inter-process communication using RDMA (remote direct memory access).&lt;br /&gt;
* All LS-Dyna versions are available in either &#039;&#039;&#039;floating point&#039;&#039;&#039; or &#039;&#039;&#039;double precision variants&#039;&#039;&#039;. Floating point variants use 4 bytes to represent a value, and double precision variants use 8 bytes. There are pros and cons for choosing one over the other variant. With regards to computational efficiency, both perform nearly the same because all machines are equipped with 64-bit CPUs.&lt;br /&gt;
** &#039;&#039;&#039;&amp;quot;f4&amp;quot;&#039;&#039;&#039; floating point versions&lt;br /&gt;
*** Pros: These require significantly less memory to run. Results occupy less disk space, and can be transferred significantly faster into and out of ARROW.&lt;br /&gt;
*** Cons: The numerical resolution is limited to 7 significant digits, which is often undesirable when dealing with mathematical operations on small and large numbers at the same time.&lt;br /&gt;
** &#039;&#039;&#039;&amp;quot;r8&amp;quot;&#039;&#039;&#039; double precision versions&lt;br /&gt;
*** Pros: The numerical resolution is about twice the number of significant digits compare to &amp;quot;f4&amp;quot;, which helps when when dealing with mathematical operations on small and large numbers at the same time.&lt;br /&gt;
*** Cons: These require more memory to run. Results occupy more disk space, and it takes longer to transfer data into and out of ARROW.&lt;br /&gt;
* There are two more identifiers to choose from when it comes to the variants of the executables: the specific compiler used to create the executable and the specific processor instruction set required for running the executable.&lt;br /&gt;
** For modern versions of LS-Dyna, two compilers have been used by the developers to create LS-Dyna executables: the &#039;&#039;&#039;Intel Fortran Compiler&#039;&#039;&#039; and the &#039;&#039;&#039;AOCC (AMD Optimizing C/C++ and Fortran) compiler&#039;&#039;&#039;. Both variants of the software are supported on ARROW. This gives users the opportunity to choose an alternate variant of the same LS-Dyna version when running into bugs or crashes.&lt;br /&gt;
** The variants based on the various instruction set extensions (&#039;&#039;&#039;SSE2&#039;&#039;&#039;, &#039;&#039;&#039;AVX2&#039;&#039;&#039;, &#039;&#039;&#039;AVX512&#039;&#039;&#039;, and so on) gives users even more options when choosing an alternate LS-Dyna variant of the same version when running into bugs or crashes. These instruction sets are mostly related to performance gains on specific processors. We have not performed thorough performance tests and cannot recommend specific versions right now.&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
$ module avail ls-dyna&lt;br /&gt;
--------------------------------------------- /shared/apps/modulefiles ---------------------------------------------&lt;br /&gt;
ls-dyna/09.3.1/1-d8-ifort131           ls-dyna/12.2.1/mpi-f4-ifort160-sse2    ls-dyna/14.2.0/mpi-f4-ifort190-avx512  &lt;br /&gt;
ls-dyna/09.3.1/1-f4-ifort131           ls-dyna/12.2.2/1-d8-ifort160-sse2      ls-dyna/14.2.0/mpi-f4-ifort190-sse2    &lt;br /&gt;
ls-dyna/09.3.1/mpi-d8-ifort131-avx2    ls-dyna/12.2.2/1-f4-ifort160-sse2      ls-dyna/15.0.2/1-d8-ifort190-sse2      &lt;br /&gt;
ls-dyna/09.3.1/mpi-d8-ifort131-avx512  ls-dyna/12.2.2/mpi-d8-aocc400-avx2     ls-dyna/15.0.2/1-f4-ifort190-sse2      &lt;br /&gt;
ls-dyna/09.3.1/mpi-f4-ifort131-avx2    ls-dyna/12.2.2/mpi-d8-ifort160-avx2    ls-dyna/15.0.2/mpi-d8-aocc400-avx2     &lt;br /&gt;
ls-dyna/09.3.1/mpi-f4-ifort131-avx512  ls-dyna/12.2.2/mpi-d8-ifort160-sse2    ls-dyna/15.0.2/mpi-d8-ifort190-avx2    &lt;br /&gt;
ls-dyna/10.2.0/1-d8-ifort160           ls-dyna/12.2.2/mpi-f4-aocc400-avx2     ls-dyna/15.0.2/mpi-d8-ifort190-avx512  &lt;br /&gt;
ls-dyna/10.2.0/1-f4-ifort160           ls-dyna/12.2.2/mpi-f4-ifort160-avx2    ls-dyna/15.0.2/mpi-d8-ifort190-sse2    &lt;br /&gt;
ls-dyna/11.0.0/1-d8-ifort160           ls-dyna/12.2.2/mpi-f4-ifort160-sse2    ls-dyna/15.0.2/mpi-f4-aocc400-avx2     &lt;br /&gt;
ls-dyna/11.0.0/1-f4-ifort160           ls-dyna/13.0.0/1-d8-ifort190           ls-dyna/15.0.2/mpi-f4-ifort190-avx2    &lt;br /&gt;
ls-dyna/11.1.0/1-d8-ifort160-sse2      ls-dyna/13.0.0/1-f4-ifort190           ls-dyna/15.0.2/mpi-f4-ifort190-avx512  &lt;br /&gt;
ls-dyna/11.1.0/1-f4-ifort160-sse2      ls-dyna/13.0.0/mpi-d8-ifort190-avx2    ls-dyna/15.0.2/mpi-f4-ifort190-sse2    &lt;br /&gt;
ls-dyna/11.2.0/1-d8-ifort160           ls-dyna/13.0.0/mpi-d8-ifort190-sse2    ls-dyna/16.0.0/1-d8-aocc420-avx2       &lt;br /&gt;
ls-dyna/11.2.0/1-f4-ifort160           ls-dyna/13.0.0/mpi-f4-ifort190-avx2    ls-dyna/16.0.0/1-d8-aocc420-avx512     &lt;br /&gt;
ls-dyna/11.2.0/mpi-f4-ifort160-avx2    ls-dyna/13.0.0/mpi-f4-ifort190-sse2    ls-dyna/16.0.0/1-d8-ifort190-sse2      &lt;br /&gt;
ls-dyna/11.2.0/mpi-f4-ifort160-sse2    ls-dyna/13.1.0/mpi-d8-aocc310-avx2     ls-dyna/16.0.0/1-f4-aocc420-avx2       &lt;br /&gt;
ls-dyna/11.2.1/1-d8-ifort160           ls-dyna/13.1.0/mpi-d8-ifort190-avx2    ls-dyna/16.0.0/1-f4-aocc420-avx512     &lt;br /&gt;
ls-dyna/11.2.1/1-f4-ifort160           ls-dyna/13.1.0/mpi-d8-ifort190-sse2    ls-dyna/16.0.0/1-f4-ifort190-sse2      &lt;br /&gt;
ls-dyna/11.2.1/mpi-d8-ifort160-avx2    ls-dyna/13.1.0/mpi-f4-aocc310-avx2     ls-dyna/16.0.0/mpi-d8-aocc420-avx2     &lt;br /&gt;
ls-dyna/11.2.1/mpi-d8-ifort160-sse2    ls-dyna/13.1.0/mpi-f4-ifort190-avx2    ls-dyna/16.0.0/mpi-d8-aocc420-avx512   &lt;br /&gt;
ls-dyna/11.2.1/mpi-f4-ifort160-avx2    ls-dyna/13.1.0/mpi-f4-ifort190-sse2    ls-dyna/16.0.0/mpi-d8-ifort190-avx2    &lt;br /&gt;
ls-dyna/11.2.1/mpi-f4-ifort160-sse2    ls-dyna/13.1.1/mpi-d8-ifort190-avx2    ls-dyna/16.0.0/mpi-d8-ifort190-avx512  &lt;br /&gt;
ls-dyna/11.2.2/mpi-d8-ifort160-avx2    ls-dyna/13.1.1/mpi-d8-ifort190-sse2    ls-dyna/16.0.0/mpi-d8-ifort190-sse2    &lt;br /&gt;
ls-dyna/11.2.2/mpi-d8-ifort160-sse2    ls-dyna/13.1.1/mpi-f4-ifort190-avx2    ls-dyna/16.0.0/mpi-f4-aocc420-avx2     &lt;br /&gt;
ls-dyna/11.2.2/mpi-f4-ifort160-avx2    ls-dyna/13.1.1/mpi-f4-ifort190-sse2    ls-dyna/16.0.0/mpi-f4-aocc420-avx512   &lt;br /&gt;
ls-dyna/11.2.2/mpi-f4-ifort160-sse2    ls-dyna/14.0.0/1-d8-ifort190           ls-dyna/16.0.0/mpi-f4-ifort190-avx2    &lt;br /&gt;
ls-dyna/12.1.0/1-d8-ifort160           ls-dyna/14.0.0/1-f4-ifort190           ls-dyna/16.0.0/mpi-f4-ifort190-avx512  &lt;br /&gt;
ls-dyna/12.1.0/1-f4-aocc310            ls-dyna/14.0.0/mpi-d8-aocc310-avx2     ls-dyna/16.0.0/mpi-f4-ifort190-sse2    &lt;br /&gt;
ls-dyna/12.1.0/1-f4-ifort160           ls-dyna/14.0.0/mpi-d8-ifort190-avx2    ls-dyna/16.1.0/mpi-d8-aocc420-avx2     &lt;br /&gt;
ls-dyna/12.1.0/mpi-d8-aocc310-avx2     ls-dyna/14.0.0/mpi-d8-ifort190-sse2    ls-dyna/16.1.0/mpi-d8-aocc420-avx512   &lt;br /&gt;
ls-dyna/12.1.0/mpi-d8-ifort160-avx2    ls-dyna/14.0.0/mpi-f4-ifort190-avx2    ls-dyna/16.1.0/mpi-d8-ifort190-avx2    &lt;br /&gt;
ls-dyna/12.1.0/mpi-d8-ifort160-sse2    ls-dyna/14.0.0/mpi-f4-ifort190-sse2    ls-dyna/16.1.0/mpi-d8-ifort190-avx512  &lt;br /&gt;
ls-dyna/12.1.0/mpi-f4-aocc310-avx2     ls-dyna/14.1.0/1-d8-ifort190-sse2      ls-dyna/16.1.0/mpi-d8-ifort190-sse2    &lt;br /&gt;
ls-dyna/12.1.0/mpi-f4-ifort160-avx2    ls-dyna/14.1.0/1-f4-ifort190-sse2      ls-dyna/16.1.0/mpi-f4-aocc420-avx2     &lt;br /&gt;
ls-dyna/12.1.0/mpi-f4-ifort160-sse2    ls-dyna/14.1.0/mpi-d8-aocc400-avx2     ls-dyna/16.1.0/mpi-f4-aocc420-avx512&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
===Submitting an LS-Dyna Job===&lt;br /&gt;
&lt;br /&gt;
&amp;lt;BLOCKQUOTE&amp;gt;&lt;br /&gt;
[[File:Attention.jpg|25px]] &#039;&#039;&#039;IMPORTANT NOTE:&#039;&#039;&#039; The job/queue manager can now track the number of LS-Dyna licenses given out to individual&lt;br /&gt;
jobs. At submission time, it is not possible to know what software a user may run. But by adding the clause &amp;quot;-l dynalic&amp;quot; at submission time,&lt;br /&gt;
the queue manager can calculate the total number of cores required and keep track of LS-Dyna licenses used by the job. When loading a version of LS-Dyna, a check will be performed, and LS-Dyna will be prevented from running if the &amp;quot;-l dynalic&amp;quot; clause was not used when submitting the job.&#039;&#039;&lt;br /&gt;
&lt;br /&gt;
&amp;lt;!--&lt;br /&gt;
&#039;&#039;The job/queue manager can track the number of LS-Dyna licenses to some degree. If all &#039;&#039;&#039;LS-Dyna users&#039;&#039;&#039; cooperate and use a script like the one shown below when submitting their jobs, the total number of concurrent &#039;&#039;&#039;LS-Dyna licenses&#039;&#039;&#039; will be tracked by the job manager correctly. That means that users can submit any number of LS-Dyna jobs, and jobs will only start when a sufficient number of licenses is available. This is managed by the &#039;&#039;&#039;&amp;quot;dynalic&amp;quot;&#039;&#039;&#039; resource at the end of the select statement. In this example, a 2-node job on 64-core nodes will need a total of &#039;&#039;&#039;&amp;quot;dynalic=128&amp;quot;&#039;&#039;&#039; licenses. This accounting breaks down when users don&#039;t use the &#039;&#039;&#039;&amp;quot;dynalic=XXX&amp;quot;&#039;&#039;&#039; statement, or when they don&#039;t calculate the number of licenses correctly. In that case, LS-Dyna jobs of all users are subject to sudden failure when LS-Dyna licenses run out. Please understand the importance of this specific setting in your job.&#039;&#039;&lt;br /&gt;
--&amp;gt;&lt;br /&gt;
&amp;lt;/BLOCKQUOTE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
Furthermore, careful consideration should be given with regards to choice of resources for an &#039;&#039;&#039;LS-Dyna job&#039;&#039;&#039;. With 64 cores available on a single node in the &#039;&#039;&#039;&amp;quot;epyc1&amp;quot;&#039;&#039;&#039; and &#039;&#039;&#039;&amp;quot;epyc2&amp;quot;&#039;&#039;&#039; queues, it may be counterproductive to run a job on two nodes instead of a single node. Users should run their jobs with different numbers of nodes and determine whether performance increases. It may well decrease when running a job on two or more nodes. The outcome of such tests will tell what the best allocation of resources will be.&lt;br /&gt;
&lt;br /&gt;
Most users use a job script like the following. All methods for job submission the the previous chapters apply as well, so there is a lot of flexibility:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
#!/bin/bash&lt;br /&gt;
#&lt;br /&gt;
#PBS -q epyc1&lt;br /&gt;
#PBS -l walltime=12:0:0&lt;br /&gt;
#PBS -l select=2:ncpus=64:mpiprocs=64:mem=225G,dynalic&lt;br /&gt;
#PBS -N JobName&lt;br /&gt;
#PBS -e log.error&lt;br /&gt;
#PBS -o log.output&lt;br /&gt;
#&lt;br /&gt;
cd $PBS_O_WORKDIR&lt;br /&gt;
#&lt;br /&gt;
module load ls-dyna/12.2.1/mpi-f4-ifort160-avx2&lt;br /&gt;
module load dynamore/current&lt;br /&gt;
#&lt;br /&gt;
mpirun ls-dyna i=main.k memory1=300m memory2=100m &amp;gt; dyna.log&lt;br /&gt;
#&lt;br /&gt;
# when using the Dynamore tools, you can start something like this at the end&lt;br /&gt;
DM.plotcprs.lnx -merge &amp;gt;&amp;gt; dyna.log&lt;br /&gt;
#&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
===LSTC Tools: LS-OPT and LS-PREPOST===&lt;br /&gt;
&lt;br /&gt;
For the new Rocky 9 cluster, I have not looked deeply into the ls-opt and ls-prepost packages that were installed. I noticed though that the LSTC server provided access to much newer versions of both software packages. If you would like to learn more or have a specific version in mind, I can easily download and install it for you.&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
$ module avail ls-opt&lt;br /&gt;
----------------------------------------------- /shared/apps/modulefiles ------------------------------------------------&lt;br /&gt;
ls-opt/5.1.1  ls-opt/6.0.0  ls-opt/7.0.0  ls-opt/7.0.2   ls-opt/2022R2  &lt;br /&gt;
ls-opt/5.2.1  ls-opt/6.1.0  ls-opt/7.0.1  ls-opt/2022R1  ls-opt/2023R1  &lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
To start the software, type:&lt;br /&gt;
 lsopt&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
$ module avail ls-prepost&lt;br /&gt;
----------------------------------------------- /shared/apps/modulefiles ------------------------------------------------&lt;br /&gt;
ls-prepost/4.5.10  ls-prepost/4.8.13  ls-prepost/4.8.30  ls-prepost/4.9.16  ls-prepost/4.10.7 &lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
To start the software, type:&lt;br /&gt;
 lsprepost&lt;br /&gt;
&lt;br /&gt;
===Dynamore Software===&lt;br /&gt;
&lt;br /&gt;
The Dynamore tools are available as a module:&lt;br /&gt;
&lt;br /&gt;
 module load dynamore/current&lt;br /&gt;
&lt;br /&gt;
We typically acquire a yearly license for the tools as we purchase licenses for LS-Dyna.&lt;br /&gt;
&lt;br /&gt;
===Vendor License File Installation===&lt;br /&gt;
&lt;br /&gt;
If you would like for us to install a vendor license for LS-Dyna models, please contact us for the required information. We can send you the general LS-Dyna license file to show the host ids for the license server. Using that information, your vendor should be able to create a vendor license file. Please send that file to us per Email or by other means.&lt;br /&gt;
&lt;br /&gt;
==StarCCM+ on the ARROW Cluster==&lt;br /&gt;
&lt;br /&gt;
===Currently Available StarCCM+ Versions===&lt;br /&gt;
&lt;br /&gt;
As of late 2025, we have the following versions of &#039;&#039;&#039;StarCCM+&#039;&#039;&#039; available on the cluster:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
$ module avail starccm&lt;br /&gt;
---------------------------- /shared/apps/modulefiles ----------------------------&lt;br /&gt;
starccm/15.02.007-R8  starccm/16.06.008-R8  starccm/18.06.006-R8  &lt;br /&gt;
starccm/15.02.009-R8  starccm/17.02.007-R8  starccm/19.02.009-R8  &lt;br /&gt;
starccm/15.04.008-R8  starccm/17.02.008-R8  starccm/20.04.007-R8  &lt;br /&gt;
starccm/15.06.008-R8  starccm/17.04.007-R8  starccm/20.06.007-R8  &lt;br /&gt;
starccm/16.02.008-R8  starccm/17.06.007-R8  &lt;br /&gt;
starccm/16.04.007-R8  starccm/18.04.008-R8&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
If using a &#039;&#039;&#039;single node&#039;&#039;&#039; for StarCCM+, job submission (for an interactive job) is simple and will use appropriate default settings:&lt;br /&gt;
&lt;br /&gt;
 qsub -I -X -q epyc1 -l walltime=20:00:00&lt;br /&gt;
&lt;br /&gt;
StarCCM+ can make use of the job scheduler attributes by automatically obtaining the number of cores and other resources from OpenPBS. In this case, the default number of cores and mpi processes for StarCCM+ are both 64 when using the epyc1 queue. So you can start your StarCCM+ run with:&lt;br /&gt;
&lt;br /&gt;
 module load starccm/15.02.007-R8 (or any other version)&lt;br /&gt;
 starccm+ -bs pbs&lt;br /&gt;
&lt;br /&gt;
In this case, there is no need for StarCCM+ to be told to run the case in parallel with the selected number of cores/mpiprocs.&lt;br /&gt;
&lt;br /&gt;
This can get a bit more complex when running on multiple nodes or when requesting high memory nodes. In that case you would use job submission parameters as shown below:&lt;br /&gt;
&lt;br /&gt;
 qsub -I -X -q epyc1 -l walltime=20:00:00,select=2:ncpus=64:mpiprocs=61:mem=500GB&lt;br /&gt;
&lt;br /&gt;
Requesting nodes that can satisfy those resources, two nodes with these attributes must exist. We have multiple nodes with 512GB in the epyc1 queue, meaning that this job will run on two machines that have at least the required amount of memory installed (on each node). The job will be queued until two machines like this will be available. If no machines with these resources exist, the job will stay in the queue forever. Therefore, you have to craft the submission string carefully.&lt;br /&gt;
&lt;br /&gt;
To accommodate high memory jobs, the nodes have been assigned priorities for assignment. Low memory jobs have the highest priority and will be assigned to nodes that can accommodate the request. High memory nodes have the lowest priority, meaning that they are the last ones given out to users. This makes it more likely that a high memory job can be run soon when the cluster is moderately loaded with jobs.&lt;br /&gt;
&lt;br /&gt;
StarCCM+ will always use the Intel MPI fabric. Other MPI versions do not work, even when selected on the command line.&lt;br /&gt;
&lt;br /&gt;
==OpenFOAM on the ARROW Cluster==&lt;br /&gt;
&lt;br /&gt;
===Currently Available OpenFOAM  Versions===&lt;br /&gt;
&lt;br /&gt;
As of late 2025, we have the following versions of OpenFOAM available on the cluster:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
$ module avail openfoam&lt;br /&gt;
------------ /shared/apps/modulefiles ------------&lt;br /&gt;
openfoam/9   openfoam/13      openfoam/v2312  &lt;br /&gt;
openfoam/10  openfoam/13-amd  openfoam/v2406  &lt;br /&gt;
openfoam/11  openfoam/v2212   &lt;br /&gt;
openfoam/12  openfoam/v2306&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
Contact us if you encounter problems; there can be various reasons why OpenFOAM may have trouble on certain hardware or when compiling dynamic code. When loading OpenFOAM modules, a number of dependencies will be automatically loaded for you, and you don&#039;t have to load those yourself. For example:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
$ module load openfoam/13&lt;br /&gt;
Loading openfoam/13&lt;br /&gt;
  Loading requirement: intel/2024.2.0/mpi/2021.13 gcc/gcc-12.1.0&lt;br /&gt;
&lt;br /&gt;
$ module list&lt;br /&gt;
Currently Loaded Modulefiles:&lt;br /&gt;
 1) intel/2024.2.0/mpi/2021.13   2) gcc/gcc-12.1.0   3) openfoam/13&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
In this case, OpenFOAM 13 loads the Intel 2024 MPI module, and loads the GCC compiler 12.1. OpenFOAM was compiled from source, and has been compiled specifically with that compiler and MPI version, so it make little sense to use other compilers or MPI libraries.&lt;br /&gt;
&lt;br /&gt;
Note: We have found a problem with running the Intel 2024 MPI library in the amd64 queue. Therefore, we have a modified module that uses the Intel 2022 library (I know -- Intel 2022 gives you the 2021 MPI libraries, but that is the way Intel distributes this software):&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
$ module load openfoam/13-amd &lt;br /&gt;
Loading mpi version 2021.7.0&lt;br /&gt;
Loading openfoam/13-amd&lt;br /&gt;
  Loading requirement: intel/2022.2.0/mpi/2021.7.0 gcc/gcc-12.1.0&lt;br /&gt;
&lt;br /&gt;
$ module list&lt;br /&gt;
Currently Loaded Modulefiles:&lt;br /&gt;
 1) intel/2022.2.0/mpi/2021.7.0   2) gcc/gcc-12.1.0   3) openfoam/13-amd&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
If you are compiling OpenFOAM yourself, the modules are of little help. You would need to select the appropriate MPI version and compiler before doing so, and then consistently load them before running your OpenFOAM executables. Within the &amp;quot;etc/bashrc&amp;quot; file in the source code tree, you want to set the MPI library to INTELMPI. As usual with self-compiled versions of OpenFOAM, you would &amp;quot;source etc/bashrc&amp;quot; to set up your personal environment to run your home-brew version of OpenFOAM. Contact us if you need to learn more about compiling OpenFOAM on the system.&lt;br /&gt;
&lt;br /&gt;
==Additional Software Applications and Libraries==&lt;br /&gt;
&lt;br /&gt;
===Loadable GCC Compiler Versions===&lt;br /&gt;
&lt;br /&gt;
The Rocky 9.6 operating system uses the GCC 11.5 compiler. That should be sufficient for most users when compiling your own applications. In case you need to use either a more up-to-date compiler, or if you need an older compiler for compatibility, we make the following versions available as loadable modules.&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
$ module avail gcc&lt;br /&gt;
------------ /shared/apps/modulefiles ------------&lt;br /&gt;
gcc/gcc-4.9.4  gcc/gcc-7.5.0  gcc/gcc-10.3.0  &lt;br /&gt;
gcc/gcc-5.5.0  gcc/gcc-8.5.0  gcc/gcc-11.3.0  &lt;br /&gt;
gcc/gcc-6.5.0  gcc/gcc-9.5.0  gcc/gcc-12.1.0&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
Additional versions can be created and made available as modules as well. If you need a specific version that is not currently available, please ask us to compiler and install it. If necessary, we may be able to provide access to other compilers, for example LLVM. We do not provide access to proprietary compilers at this time.&lt;br /&gt;
&lt;br /&gt;
===MPI Libraries and Runtimes===&lt;br /&gt;
&lt;br /&gt;
While we seem to have a variety of MPI versions and flavors available to users, the only MPI versions that allow us to run software over Infiniband are the Intel MPI libraries. Some of the installed alternatives are likely to fail, or will have a set of environment variables that have to be set. All major engineering software packages that we offer are pre-configured with specific MPI versions and settings that have been tested and/or provided by the vendors.&lt;br /&gt;
&lt;br /&gt;
Note: Some MPI libraries may seem to work. They may allow your MPI application to run. But inter-process network communication may travel through the rather slow and high-latency Ethernet fabric, making MPI applications very ineffective and are probably not worth while.&lt;br /&gt;
&lt;br /&gt;
===MatLab Runtimes===&lt;br /&gt;
&lt;br /&gt;
We can install MatLAB run time libraries as needed and have them available as loadable modules. Recently, we had a problem with MatLAB run time libraries being identified as security vulnerabilities. Contact us if you need them installed for one of your projects.&lt;br /&gt;
&lt;br /&gt;
===Anaconda and variants (miniconda etc)===&lt;br /&gt;
&lt;br /&gt;
Our current practice is to have users download and install their own versions of Anaconda and its variants in their own home directories. This allows for maximum flexibility when it comes to installable software modules, and users can maintain the installation, upgrades, and maintenance themselves. If you encounter issues, please contact us. One known side effect of Anaconda installations is a performance hit when starting your software, e.g. python scripts may take 30 seconds or more to execute. This is an artefact caused by the Lustre file system, which has been designed for large files accessible from many machines simultaneously. Performance on reading many small files has not been considered and is fairly poor. Again, contact us and we will design a solution for you as needed.&lt;/div&gt;</summary>
		<author><name>Ley</name></author>
	</entry>
	<entry>
		<id>https://wiki.anl.gov/wiki_tracc/index.php?title=Job_Submission_and_Monitoring&amp;diff=2490</id>
		<title>Job Submission and Monitoring</title>
		<link rel="alternate" type="text/html" href="https://wiki.anl.gov/wiki_tracc/index.php?title=Job_Submission_and_Monitoring&amp;diff=2490"/>
		<updated>2026-02-25T16:52:52Z</updated>

		<summary type="html">&lt;p&gt;Ley: /* LSTC Tools: LS-OPT and LS-PREPOST */&lt;/p&gt;
&lt;hr /&gt;
&lt;div&gt;{| align=&amp;quot;right&amp;quot;&lt;br /&gt;
| __TOC__&lt;br /&gt;
|}&lt;br /&gt;
==Resource Summary View==&lt;br /&gt;
&lt;br /&gt;
To get started, users can query the overall status of resources on the cluster. The &#039;&#039;&#039;&amp;quot;qsum&amp;quot;&#039;&#039;&#039; script will list all queues and nodes, as well as how many are offline, down, free, or assigned to users. This is a script developed by our team, and may need to be updated if something goes wrong. Please contact us if you experience any problems.&lt;br /&gt;
&lt;br /&gt;
Each queue groups a number of nodes together based on their hardware and software configurations. Nodes can be part of more than one queue, and there are other complex details that we are ignoring here for the purpose of keeping it simple.&lt;br /&gt;
&lt;br /&gt;
===Queues===&lt;br /&gt;
&lt;br /&gt;
Here is a very brief summary of what each of the queues is, and how to use them efficiently:&lt;br /&gt;
&lt;br /&gt;
; a4000: This is a queue that has three 16-core CPU machines, each of which is furthermore equipped with three A4000 GPUs. That makes a total of 9 A4000 GPUs available to users. Neither the GPUs nor the processors are particularly powerful these days. The machines have 500GB of memory though, which makes for a good platform for experimenting with GPU capabilities.&lt;br /&gt;
; a6000: This is a queue that has only one 64-core CPU machines, and is equipped with four A6000 GPUs. The system can be upgraded to 8 A6000 GPUs if needed. This is a decent GPU machine that can take a solid workload these days. The machine has 750GB of memory, which makes for a good production platform.&lt;br /&gt;
; amd16: This is a queue with many of our older AMD-based 16-core machines, each of which has 30GB of memory. While individual machines are a bit outdated, they are all interconnected with Infiniband and can provide a solid production workload in multi-nodes jobs over MPI without blocking the more current (and thus expensive) systems.&lt;br /&gt;
; epyc1/epyc2: These are 2 separate queues with slightly different performance characteristics. Each of the groups is interconnected with Infiniband to provide a platform for large and demanding software packages, such as LS-Dyna and StarCCM+. They have between 250GB and 500GB of memory. Because licenses for these software packages are very expensive, they should use these two queues for making optimum use of limited core licenses available to each package.&lt;br /&gt;
; xeon28: This is a set of intermediate machines with 28 cores and 64GB of memory. They can be used for a variety of purposes, including MPI jobs and single node application software.&lt;br /&gt;
; virtual: This is a set of nodes without MPI capabilities. They are virtual machines with 32GB each. They can be used for higher demand applications that would interfere with the login nodes, and therefore with other users of these login machines. A user would submit interactive jobs to individual virtual machines and avoid any significant load on login nodes.&lt;br /&gt;
&lt;br /&gt;
===The Queue Summary Script (qsum)===&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
$ qsum&lt;br /&gt;
=============== a4000 ==========================================================&lt;br /&gt;
Queue: &amp;quot;a4000&amp;quot; / nodes: 3 / down: 0 / offline: 0 / busy: 0 / available: 3&lt;br /&gt;
      AVAILABLE (3): g001, g002, g003&lt;br /&gt;
=============== a6000 ==========================================================&lt;br /&gt;
Queue: &amp;quot;a6000&amp;quot; / nodes: 1 / down: 0 / offline: 0 / busy: 0 / available: 1&lt;br /&gt;
      AVAILABLE (1): lambda01&lt;br /&gt;
=============== amd16 ==========================================================&lt;br /&gt;
Queue: &amp;quot;amd16&amp;quot; / nodes: 33 / down: 2 / offline: 0 / busy: 2 / available: 29&lt;br /&gt;
           DOWN (2): n017, n030&lt;br /&gt;
            ley (2): n001, n002&lt;br /&gt;
     AVAILABLE (29): n003, n004, n005, n006, n007, n008, n009, n010, n011, n012&lt;br /&gt;
                     n013, n014, n015, n016, n018, n019, n020, n021, n022, n023&lt;br /&gt;
                     n024, n025, n026, n027, n028, n029, n031, n032, n039&lt;br /&gt;
=============== epyc1 ==========================================================&lt;br /&gt;
Queue: &amp;quot;epyc1&amp;quot; / nodes: 1 / down: 0 / offline: 0 / busy: 0 / available: 1&lt;br /&gt;
      AVAILABLE (1): a027&lt;br /&gt;
=============== epyc2 ==========================================================&lt;br /&gt;
Queue: &amp;quot;epyc2&amp;quot; / nodes: 20 / down: 0 / offline: 0 / busy: 5 / available: 15&lt;br /&gt;
            ley (2): a030, a031&lt;br /&gt;
         msitek (3): a028, a029, a032&lt;br /&gt;
     AVAILABLE (15): a033, a034, a035, a036, a037, a038, a039, a040, a041, a042&lt;br /&gt;
                     a043, a044, a045, a046, a047&lt;br /&gt;
=============== virtual ========================================================&lt;br /&gt;
Queue: &amp;quot;virtual&amp;quot; / nodes: 6 / down: 0 / offline: 0 / busy: 0 / available: 6&lt;br /&gt;
      AVAILABLE (6): v001, v002, v003, v004, v005, v006&lt;br /&gt;
=============== xeon28 =========================================================&lt;br /&gt;
Queue: &amp;quot;xeon28&amp;quot; / nodes: 12 / down: 0 / offline: 0 / busy: 0 / available: 12&lt;br /&gt;
     AVAILABLE (12): p001, p002, p003, p004, p005, p006, p007, p008, p009, p010&lt;br /&gt;
                     p011, p012&lt;br /&gt;
================================================================================&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
==Queue Status and Monitoring Jobs==&lt;br /&gt;
&lt;br /&gt;
===qstat===&lt;br /&gt;
&lt;br /&gt;
To find out out about the status of all running jobs on the cluster you can use the &#039;&#039;&#039;&amp;quot;qstat&amp;quot;&#039;&#039;&#039; command. Here is an example:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
$ qstat&lt;br /&gt;
&lt;br /&gt;
Nov 20 18:30 ley@login3:Plots$ qstat&lt;br /&gt;
Job id            Name             User              Time Use S Queue&lt;br /&gt;
----------------  ---------------- ----------------  -------- - -----&lt;br /&gt;
3023.pbs          STDIN            msitek            4144:14* R epyc2           &lt;br /&gt;
3029.pbs          STDIN            ley               76:46:53 R epyc2           &lt;br /&gt;
3032.pbs          STDIN            msitek            2879:52* R epyc2           &lt;br /&gt;
3033.pbs          STDIN            msitek            3687:29* R epyc2           &lt;br /&gt;
3048.pbs          foo.sh           james.cook               0 Q amd16           &lt;br /&gt;
3060.pbs          of13.sh          ley               310:47:* R epyc2           &lt;br /&gt;
3061.pbs          of13.sh          ley               308:37:* R epyc2           &lt;br /&gt;
3062.pbs          of13.sh          ley               308:02:* R epyc2           &lt;br /&gt;
3063.pbs          of13.sh          ley               308:15:* R epyc2&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
The first column shows the &#039;&#039;&#039;job id&#039;&#039;&#039;, a unique identifier for all jobs ever submitted to the cluster. This job id is important when killing jobs, or for other actions you may need to take.&lt;br /&gt;
&lt;br /&gt;
The next column shows the name of the job script. If the column shows &#039;&#039;&#039;STDIN&#039;&#039;&#039;, it means that this is an &#039;&#039;&#039;interactive job&#039;&#039;&#039; where a user can enter commands in a terminal window. This is particularly useful for model and software development task where the application has to be started and killed repeatedly.&lt;br /&gt;
&lt;br /&gt;
The owner of the job is shown next. These are the user names of the various people using the cluster.&lt;br /&gt;
&lt;br /&gt;
The last three columns indicate the current run time of the job, whether it is running (R) or waiting (Q) for execution. The last entry shows the queue in which the job is running.&lt;br /&gt;
&lt;br /&gt;
===qstat -an1===&lt;br /&gt;
&lt;br /&gt;
Adding a few options gives much more detail about each jobs.&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
qstat -an1&lt;br /&gt;
&lt;br /&gt;
Nov 20 13:09 ley@login3:Plots$ qstat -an1&lt;br /&gt;
&lt;br /&gt;
pbs: &lt;br /&gt;
                                                            Req&#039;d  Req&#039;d   Elap&lt;br /&gt;
Job ID          Username Queue    Jobname    SessID NDS TSK Memory Time  S Time&lt;br /&gt;
--------------- -------- -------- ---------- ------ --- --- ------ ----- - -----&lt;br /&gt;
3023.pbs        msitek   epyc2    STDIN      24360*   1  64  350gb 100:0 R 81:46 a028/0*64&lt;br /&gt;
3029.pbs        ley      epyc2    STDIN      21719*   2 128  100gb 200:0 R 72:31 a030/0*64+a031/0*64&lt;br /&gt;
3032.pbs        msitek   epyc2    STDIN      18102*   1  64  350gb 100:0 R 57:57 a029/0*64&lt;br /&gt;
3033.pbs        msitek   epyc2    STDIN      830486   1  64  350gb 100:0 R 57:53 a032/0*64&lt;br /&gt;
3048.pbs        james.c* amd16    foo.sh        --    1  28   30gb 01:00 Q   --   -- &lt;br /&gt;
3060.pbs        ley      epyc2    STDIN      763101   1  64  350gb 48:00 R 06:42 a033/0*64&lt;br /&gt;
3061.pbs        ley      epyc2    STDIN      763947   1  64  350gb 48:00 R 06:40 a034/0*64&lt;br /&gt;
3062.pbs        ley      epyc2    STDIN      761473   1  64  350gb 48:00 R 06:39 a035/0*64&lt;br /&gt;
3063.pbs        ley      epyc2    STDIN      766205   1  64  350gb 48:00 R 06:40 a036/0*64&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
In this table, you can see how many nodes and cores are being used by each job. For example, job 3029 of the user &amp;quot;ley&amp;quot; shows that it is running on 2 nodes using a total of 128 cores. In addition to the elapsed time, the table also show the reserved time for this job. This allow you to estimate when a job will be definitely finalized (or killed by the system if still running).&lt;br /&gt;
&lt;br /&gt;
The last column (without a header) is written because the option &#039;&#039;&#039;&amp;quot;-an1&amp;quot;&#039;&#039;&#039; was used. This useful to learn about which nodes are used by each job.&lt;br /&gt;
&lt;br /&gt;
===qstat -q===&lt;br /&gt;
&lt;br /&gt;
To learn more about the queues on the cluster, the &#039;&#039;&#039;&amp;quot;-q&amp;quot;&#039;&#039;&#039; option turns out to be useful.&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
$ qstat -q&lt;br /&gt;
&lt;br /&gt;
server: pbs&lt;br /&gt;
&lt;br /&gt;
Queue            Memory CPU Time Walltime Node   Run   Que   Lm  State&lt;br /&gt;
---------------- ------ -------- -------- ---- ----- ----- ----  -----&lt;br /&gt;
virtual            30gb    --       --       1     0     0   --   E R&lt;br /&gt;
a4000             500gb    --       --       1     0     0   --   E R&lt;br /&gt;
a6000             750gb    --       --       1     0     0   --   E R&lt;br /&gt;
xeon28             --      --       --       4     0     0   --   E R&lt;br /&gt;
amd16              --      --       --       8     0     1   --   E R&lt;br /&gt;
epyc2              --      --       --       2    14     0   --   E R&lt;br /&gt;
epyc1              --      --       --       2     0     0   --   E R&lt;br /&gt;
                                               ----- -----&lt;br /&gt;
                                                  14     1&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
For each queue, some basic values are displayed. The first three queues listed above have a default memory allocation as shown, and the column &#039;&#039;&#039;&amp;quot;Node&amp;quot;&#039;&#039;&#039; indicates the maximum number of nodes that can be asked for at job submission time. For example, you can request just one node for a job from the first three queues (because these are queues where MPI makes no sense). The &#039;&#039;&#039;&amp;quot;xeon28&amp;quot;&#039;&#039;&#039; queue also for a maximum of 4 nodes per MPI job. The &#039;&#039;&#039;&amp;quot;amd16&amp;quot;&#039;&#039;&#039; queue has a maximum of 8 nodes per job, and the &#039;&#039;&#039;&amp;quot;epyc1&amp;quot;&#039;&#039;&#039; and &#039;&#039;&#039;&amp;quot;epyc2&amp;quot;&#039;&#039;&#039; queues have maxima of two nodes per job. These limitations can be changed by the administrator as needed. As shown above, this will prevent inefficient resource requests.&lt;br /&gt;
&lt;br /&gt;
===qstat -f===&lt;br /&gt;
&lt;br /&gt;
The command &#039;&#039;&#039;&amp;quot;qstat -f -F json 3029&amp;quot;&#039;&#039;&#039; retrieves extremely detailed stats on the running job 3029. The result can be returned in JSON format to be ready for further processing (shown below).&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
$ qstat -f -F json 3029&lt;br /&gt;
{&lt;br /&gt;
    &amp;quot;timestamp&amp;quot;:1763705353,&lt;br /&gt;
    &amp;quot;pbs_version&amp;quot;:&amp;quot;23.06.06&amp;quot;,&lt;br /&gt;
    &amp;quot;pbs_server&amp;quot;:&amp;quot;pbs&amp;quot;,&lt;br /&gt;
    &amp;quot;Jobs&amp;quot;:{&lt;br /&gt;
        &amp;quot;3029.pbs&amp;quot;:{&lt;br /&gt;
            &amp;quot;Job_Name&amp;quot;:&amp;quot;STDIN&amp;quot;,&lt;br /&gt;
            &amp;quot;Job_Owner&amp;quot;:&amp;quot;ley@login4&amp;quot;,&lt;br /&gt;
            &amp;quot;resources_used&amp;quot;:{&lt;br /&gt;
                &amp;quot;cpupercent&amp;quot;:98,&lt;br /&gt;
                &amp;quot;cput&amp;quot;:&amp;quot;76:46:53&amp;quot;,&lt;br /&gt;
                &amp;quot;hpmem&amp;quot;:&amp;quot;0b&amp;quot;,&lt;br /&gt;
                &amp;quot;mem&amp;quot;:&amp;quot;52428800kb&amp;quot;,&lt;br /&gt;
                &amp;quot;ncpus&amp;quot;:128,&lt;br /&gt;
                &amp;quot;vmem&amp;quot;:&amp;quot;52428800kb&amp;quot;,&lt;br /&gt;
                &amp;quot;walltime&amp;quot;:&amp;quot;78:09:32&amp;quot;&lt;br /&gt;
            },&lt;br /&gt;
            &amp;quot;job_state&amp;quot;:&amp;quot;R&amp;quot;,&lt;br /&gt;
            &amp;quot;queue&amp;quot;:&amp;quot;epyc2&amp;quot;,&lt;br /&gt;
            &amp;quot;server&amp;quot;:&amp;quot;pbs&amp;quot;,&lt;br /&gt;
            &amp;quot;Checkpoint&amp;quot;:&amp;quot;u&amp;quot;,&lt;br /&gt;
            &amp;quot;ctime&amp;quot;:&amp;quot;Mon Nov 17 17:58:25 2025&amp;quot;,&lt;br /&gt;
            &amp;quot;Error_Path&amp;quot;:&amp;quot;/dev/pts/0&amp;quot;,&lt;br /&gt;
            &amp;quot;exec_host&amp;quot;:&amp;quot;a030/0*64+a031/0*64&amp;quot;,&lt;br /&gt;
            &amp;quot;exec_vnode&amp;quot;:&amp;quot;(a030:ncpus=64:mem=52428800kb)+(a031:ncpus=64:mem=52428800kb)&amp;quot;,&lt;br /&gt;
            &amp;quot;Hold_Types&amp;quot;:&amp;quot;n&amp;quot;,&lt;br /&gt;
            &amp;quot;interactive&amp;quot;:&amp;quot;True&amp;quot;,&lt;br /&gt;
            &amp;quot;Join_Path&amp;quot;:&amp;quot;n&amp;quot;,&lt;br /&gt;
            &amp;quot;Keep_Files&amp;quot;:&amp;quot;n&amp;quot;,&lt;br /&gt;
            &amp;quot;Mail_Points&amp;quot;:&amp;quot;a&amp;quot;,&lt;br /&gt;
            &amp;quot;mtime&amp;quot;:&amp;quot;Fri Nov 21 00:07:59 2025&amp;quot;,&lt;br /&gt;
            &amp;quot;Output_Path&amp;quot;:&amp;quot;/dev/pts/0&amp;quot;,&lt;br /&gt;
            &amp;quot;Priority&amp;quot;:0,&lt;br /&gt;
            &amp;quot;qtime&amp;quot;:&amp;quot;Mon Nov 17 17:58:25 2025&amp;quot;,&lt;br /&gt;
            &amp;quot;Rerunable&amp;quot;:&amp;quot;False&amp;quot;,&lt;br /&gt;
            &amp;quot;Resource_List&amp;quot;:{&lt;br /&gt;
                &amp;quot;mem&amp;quot;:&amp;quot;100gb&amp;quot;,&lt;br /&gt;
                &amp;quot;mpiprocs&amp;quot;:128,&lt;br /&gt;
                &amp;quot;ncpus&amp;quot;:128,&lt;br /&gt;
                &amp;quot;nodect&amp;quot;:2,&lt;br /&gt;
                &amp;quot;place&amp;quot;:&amp;quot;free&amp;quot;,&lt;br /&gt;
                &amp;quot;select&amp;quot;:&amp;quot;2:ncpus=64:mem=50gb:mpiprocs=64&amp;quot;,&lt;br /&gt;
                &amp;quot;walltime&amp;quot;:&amp;quot;200:00:00&amp;quot;&lt;br /&gt;
            },&lt;br /&gt;
            &amp;quot;stime&amp;quot;:&amp;quot;Mon Nov 17 17:58:25 2025&amp;quot;,&lt;br /&gt;
            &amp;quot;session_id&amp;quot;:2171964,&lt;br /&gt;
            &amp;quot;jobdir&amp;quot;:&amp;quot;/mnt/lustre/arrow/home/ley&amp;quot;,&lt;br /&gt;
            &amp;quot;substate&amp;quot;:42,&lt;br /&gt;
            &amp;quot;Variable_List&amp;quot;:{&lt;br /&gt;
                &amp;quot;PBS_O_HOME&amp;quot;:&amp;quot;/mnt/lustre/arrow/home/ley&amp;quot;,&lt;br /&gt;
                &amp;quot;PBS_O_LANG&amp;quot;:&amp;quot;en_US.UTF-8&amp;quot;,&lt;br /&gt;
                &amp;quot;PBS_O_LOGNAME&amp;quot;:&amp;quot;ley&amp;quot;,&lt;br /&gt;
                &amp;quot;PBS_O_PATH&amp;quot;:&amp;quot;/shared/apps/active/lstc/lsprepost/SP-4.5:/shared/apps/active/lstc/lsprepost/DP-4.3:/shared/bin:/usr/share/Modules/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/pbs/bin:/opt/thinlinc/bin:/opt/thinlinc/sbin:/mnt/lustre/arrow/home/ley/.local/bin:/mnt/lustre/arrow/home/ley/bin&amp;quot;,&lt;br /&gt;
                &amp;quot;PBS_O_MAIL&amp;quot;:&amp;quot;/var/spool/mail/ley&amp;quot;,&lt;br /&gt;
                &amp;quot;PBS_O_SHELL&amp;quot;:&amp;quot;/bin/bash&amp;quot;,&lt;br /&gt;
                &amp;quot;PBS_O_WORKDIR&amp;quot;:&amp;quot;/mnt/lustre/arrow/home/ley/Qualification/LS-Dyna/Rocky9/Seatbelt/Template&amp;quot;,&lt;br /&gt;
                &amp;quot;PBS_O_SYSTEM&amp;quot;:&amp;quot;Linux&amp;quot;,&lt;br /&gt;
                &amp;quot;PBS_O_QUEUE&amp;quot;:&amp;quot;epyc2&amp;quot;,&lt;br /&gt;
                &amp;quot;PBS_O_HOST&amp;quot;:&amp;quot;login4&amp;quot;&lt;br /&gt;
            },&lt;br /&gt;
            &amp;quot;comment&amp;quot;:&amp;quot;Job run at Mon Nov 17 at 17:58 on (a030:ncpus=64:mem=52428800kb)+(a031:ncpus=64:mem=52428800kb)&amp;quot;,&lt;br /&gt;
            &amp;quot;etime&amp;quot;:&amp;quot;Mon Nov 17 17:58:25 2025&amp;quot;,&lt;br /&gt;
            &amp;quot;run_count&amp;quot;:1,&lt;br /&gt;
            &amp;quot;Submit_arguments&amp;quot;:&amp;quot;-I -q epyc2 -l walltime=200:00:00,select=2:ncpus=64:mem=50gb:mpiprocs=64&amp;quot;,&lt;br /&gt;
            &amp;quot;project&amp;quot;:&amp;quot;_pbs_project_default&amp;quot;,&lt;br /&gt;
            &amp;quot;Submit_Host&amp;quot;:&amp;quot;login4&amp;quot;&lt;br /&gt;
        }&lt;br /&gt;
    }&lt;br /&gt;
}&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
===Manual pages for qstat===&lt;br /&gt;
&lt;br /&gt;
To learn more about the &#039;&#039;&#039;&amp;quot;qstat&amp;quot;&#039;&#039;&#039; command, you can use the command &#039;&#039;&#039;&amp;quot;man qstat&amp;quot;&#039;&#039;&#039;, which will print a lot of detailed information about the capabilities of this command.&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
$ man qstat&lt;br /&gt;
&lt;br /&gt;
qstat(1B)                                       PBS Professional                                       qstat(1B)&lt;br /&gt;
&lt;br /&gt;
NAME&lt;br /&gt;
       qstat - display status of PBS jobs, queues, or servers&lt;br /&gt;
&lt;br /&gt;
SYNOPSIS&lt;br /&gt;
       Displaying Job Status&lt;br /&gt;
              Default format:&lt;br /&gt;
              qstat [-E] [-J] [-p] [-t] [-w] [-x] [[&amp;lt;job ID&amp;gt; | &amp;lt;destination&amp;gt;] ...]&lt;br /&gt;
&lt;br /&gt;
              Long format:&lt;br /&gt;
              qstat -f [-F json | dsv [-D &amp;lt;delimiter&amp;gt;]] [-E] [-J] [-p] [-t] [-w]&lt;br /&gt;
                    [-x] [[&amp;lt;job ID&amp;gt; | &amp;lt;destination&amp;gt;] ...]&lt;br /&gt;
... &amp;lt;many more pages&amp;gt;&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
==Job Submission Basics==&lt;br /&gt;
&lt;br /&gt;
Jobs are submitted into the system using the &#039;&#039;&#039;&amp;quot;qsub&amp;quot;&#039;&#039;&#039; application. This application can take many different options and allows for a lot of different resource requests to tell the cluster what to do. We are running &#039;&#039;&#039;OpenPBS 23.06.06&#039;&#039;&#039; as our job scheduler. Here is a link to the User&#039;s Manual (of PBS PRO) if you want to explore gory details and capabilities. The User&#039;s Guide has about 240 pages, the Reference Guide has 500 pages, and the Big Book has 2500 pages. So there is a lot of information available. I also added job submission info for the LCRC cluster.&lt;br /&gt;
&lt;br /&gt;
* [https://argonne-lcrc.github.io/user-guides/running-jobs-at-lcrc/pbs-pro/ Argonne&#039;s LCRC pages on job submissions on their clusters]&lt;br /&gt;
* [https://help.altair.com/2022.1.0/PBS%20Professional/PBSUserGuide2022.1.pdf PBS Professional 2022.1 User&#039;s Guide]&lt;br /&gt;
* [https://help.altair.com/2022.1.0/PBS%20Professional/PBSReferenceGuide2022.1.pdf PBS Professional 2022.1 Reference Guide]&lt;br /&gt;
* [https://help.altair.com/2022.1.0/PBS%20Professional/PBS2022.1.pdf Altair PBS Professional 2022.1 Big Book]&lt;br /&gt;
&lt;br /&gt;
The User&#039;s Guide can be very helpful to clarify some of the concepts and capabilities, but it can be hard to find the specific information you may be looking for. Please understand that we are no longer running TORQUE and MAUI, so the syntax for job submission is distinctively different yet quite similar.&lt;br /&gt;
&lt;br /&gt;
The reference guide may be helpful to understand the complete syntax and full capabilities of the software.&lt;br /&gt;
&lt;br /&gt;
The big book is what I had to use when configuring OpenPBS earlier this year. This includes all the tricky details needed to make the system work smoothly for us. It&#039;s a bit scary to look at a PDF file that is 2500 pages long, but that is nothing compared to the StarCCM+ manuals.&lt;br /&gt;
&lt;br /&gt;
&amp;lt;BLOCKQUOTE&amp;gt;&lt;br /&gt;
[[File:Attention.jpg|25px]] &#039;&#039;&#039;IMPORTANT NOTE:&#039;&#039;&#039; &#039;&#039;The following sections are important to understand. They explain how jobs are submitted and then scjeduled for execution based on resources available and the specific need of the user.&#039;&#039;&lt;br /&gt;
&amp;lt;/BLOCKQUOTE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
The following sections explain the various tasks you may want to submit fir execution, ordered from simple to complex.&lt;br /&gt;
&lt;br /&gt;
* General Batch Jobs&lt;br /&gt;
** Requesting a Single Node for a Job&lt;br /&gt;
** Requesting Multiple Nodes for a Job&lt;br /&gt;
* Embedded Job Resource Requests&lt;br /&gt;
* Interactive Jobs&lt;br /&gt;
* Interactive Jobs with X-Windows GUI Applications&lt;br /&gt;
* Running Multiple Jobs on Single Nodes&lt;br /&gt;
* Running Jobs using GPUs&lt;br /&gt;
&lt;br /&gt;
===General Batch Jobs===&lt;br /&gt;
&lt;br /&gt;
Let&#039;s get started with a very basic usage of the system. Let&#039;s assume you have a simple application, and you want to execute it on a cluster node. Let&#039;s also assume that this is a very simple application, one that runs on one or a few cores, doesn&#039;t require any keyboard interaction with the user, doesn&#039;t need the user to see what&#039;s typically written to the screen, and writes its output to files. In this case, we can submit this application as a batch job, which will place it into an execution queue and process it as soon as a node becomes available.&lt;br /&gt;
&lt;br /&gt;
If the application requires more cores than a single node can provide, we can run the application over Infiniband with MPI message passing. In this case, we need to understand the concept of MPI applications a bit better. In both cases, we get started by creating a folder on the file system. Naming conventions are important so that you can distinguish the jobs by folder name.&lt;br /&gt;
&lt;br /&gt;
For both of the above scenarios, you would typically create a Bash shell script, and then submit the script into one of the queues for eventual execution.&lt;br /&gt;
&lt;br /&gt;
====Requesting a Single Node for a Job====&lt;br /&gt;
&lt;br /&gt;
Let&#039;s try something rather trivial to get used to the concept. Create yourself a folder, for example &#039;&#039;&#039;&amp;quot;myjobfolder&amp;quot;&#039;&#039;&#039;. Within that folder, create a job submission script. That script can have any name, but something short and simple may be best. Let&#039;s assume you create a file called &#039;&#039;&#039;&amp;quot;cluster.job&amp;quot;&#039;&#039;&#039;. The file doesn&#039;t have to have that extension. Any file name will do. But using the same filename for all of your jobs helps finding your way around the many files that will be created over time. The &#039;&#039;&#039;&amp;quot;cluster.job&amp;quot;&#039;&#039;&#039; file should look something like this:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
#!/bin/bash&lt;br /&gt;
#&lt;br /&gt;
# the following ensures that you will change into the directory where you are&lt;br /&gt;
# submitting the job from.&lt;br /&gt;
cd $PBS_O_WORKDIR&lt;br /&gt;
#&lt;br /&gt;
# now we sleep for 60 seconds and waste time. This is a placeholder for your application,&lt;br /&gt;
# which would be doing useful work for you.&lt;br /&gt;
sleep 60&lt;br /&gt;
#&lt;br /&gt;
# and after doing things, we may want to write something into a file to show that&lt;br /&gt;
# our jobs is done.&lt;br /&gt;
echo `date` &amp;gt; info.log&lt;br /&gt;
#&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
This can be submitted without detailed resource specifications (except for the walltime, which is be default 0:00:00):&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
$ qsub -q virtual -l walltime=1:00:00 cluster.job &lt;br /&gt;
3072.pbs&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
Wait a little, then check the status of running jobs:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
$ qstat -an1&lt;br /&gt;
&lt;br /&gt;
pbs: &lt;br /&gt;
                                                            Req&#039;d  Req&#039;d   Elap&lt;br /&gt;
Job ID          Username Queue    Jobname    SessID NDS TSK Memory Time  S Time&lt;br /&gt;
--------------- -------- -------- ---------- ------ --- --- ------ ----- - -----&lt;br /&gt;
3023.pbs        msitek   epyc2    STDIN      24360*   1  64  350gb 100:0 R 83:17 a028/0*64&lt;br /&gt;
3029.pbs        ley      epyc2    STDIN      21719*   2 128  100gb 200:0 R 74:00 a030/0*64+a031/0*64&lt;br /&gt;
3033.pbs        msitek   epyc2    STDIN      830486   1  64  350gb 100:0 R 59:23 a032/0*64&lt;br /&gt;
3048.pbs        james.c* amd16    foo.sh        --    1  28   30gb 01:00 Q   --   -- &lt;br /&gt;
3060.pbs        ley      epyc2    STDIN      763101   1  64  350gb 48:00 R 08:10 a033/0*64&lt;br /&gt;
3061.pbs        ley      epyc2    STDIN      763947   1  64  350gb 48:00 R 08:10 a034/0*64&lt;br /&gt;
3070.pbs        ley      epyc2    STDIN      766847   1  64  350gb 48:00 R 07:23 a042/0*64&lt;br /&gt;
3072.pbs        ley      virtual  cluster.j* 230230   1   4   30gb 01:00 E 00:01 v001/0*4&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
In this particular example, we are sending this job to the &#039;&#039;&#039;queue &amp;quot;virtual&amp;quot;&#039;&#039;&#039;. This queue, by default, allocates 30GB of memory to the job, and runs on 1 node with 4 cores. This is sufficient capacity to run quite a workload. When submitting a job to a single node, &#039;&#039;&#039;reasonable maximum allocations are automatically assigned&#039;&#039;&#039;, and the user doesn&#039;t have to worry about running out of memory or how many cores he will be using.&lt;br /&gt;
&lt;br /&gt;
The only required argument is the &#039;&#039;&#039;&amp;quot;walltime&amp;quot;&#039;&#039;&#039; argument. By default, the job will quit as soon as it is submitted. This indicates to the user that he forgot to provide the &#039;&#039;&#039;&amp;quot;walltime&amp;quot;&#039;&#039;&#039; argument.&lt;br /&gt;
&lt;br /&gt;
When the job disappears from the job list, it is done. At this point, you will find the file &amp;quot;info.log&amp;quot; in your job folder.&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
$ cat info.log &lt;br /&gt;
Thu Nov 20 08:00:31 PM CST 2025&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
====Requesting Multiple Nodes for a Job====&lt;br /&gt;
&lt;br /&gt;
To run jobs on multiple nodes, you will be likely &#039;&#039;&#039;executing jobs using MPI&#039;&#039;&#039;, the message passing interface. This establishes high-speed low-latency interconnections between the cores on one machine and the cores on the other machines. Data transfer does not require involvement of the cores themselves. Instead, the core tell the InfiniBand interconnect (and cores on the same node through shared memory) to transfer the data through RDMA, remoted direct memory access. The cores don&#039;t need to spend CPU cycles on copying data, but rather simply access the data once it has been copied by the Infiniband fabric. This makes for extremely efficient remote memory access, and message passing is used to coordinate data transfer between the cores no matter where they are located on any of the nodes.&lt;br /&gt;
&lt;br /&gt;
On our cluster, MPI-aware applications like &#039;&#039;&#039;OpenFOAM&#039;&#039;&#039;, &#039;&#039;&#039;StarCCM+&#039;&#039;&#039;, and &#039;&#039;&#039;LS-Dyna&#039;&#039;&#039; can be loaded as modules, which then automatically selects the most appropriate MPI library to use. The software applications have been tested to ensure that they work out-of-the box if a user selects any specific version of any of the applications.&lt;br /&gt;
&lt;br /&gt;
The following is a very trivial example for the MPI execution of a very simple executable, with one copy running on each core of the nodes allocated to  the job. It doesn&#039;t perform any real work and just wastes resources for a short time, but it illustrates how execution on the cores of various nodes works.&lt;br /&gt;
&lt;br /&gt;
Like in the previous section, we start with a simple job script that we submit to an appropriate queues. In this case, we pick a queue that has machines with Infiniband interfaces supporting efficient communications. Let&#039;s assume we edit a file with the name &#039;&#039;&#039;&amp;quot;parallel.job&amp;quot;&#039;&#039;&#039; like this:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
#!/bin/bash&lt;br /&gt;
#&lt;br /&gt;
# the following ensures that you will change into the directory where you are&lt;br /&gt;
# submitting the job from.&lt;br /&gt;
cd $PBS_O_WORKDIR&lt;br /&gt;
#&lt;br /&gt;
# to execute a simple command on all of the cores of all of the nodes allocated to the job,&lt;br /&gt;
# we need to make one of the MPI versions available. Let&#039;s use one of the most up-to-date&lt;br /&gt;
# MPI library available on the cluster&lt;br /&gt;
module load intel/2024.2.0/mpi/2021.13&lt;br /&gt;
#&lt;br /&gt;
# now we are apply a few settings that ensure that the MPI library will use the highest-performing&lt;br /&gt;
# Infiniband Interconnect, as well as a few options to tell MPI how to interface nodes with&lt;br /&gt;
# each other and which specific Infiniband adapter to use. This is complex and requires in-depth&lt;br /&gt;
# knowledge of the QLogic Infiniband adapters we are using. It is unlikely that you will ever have to&lt;br /&gt;
# deal with these options, because the &amp;quot;module load&amp;quot; command for the engineering applications we provide&lt;br /&gt;
# on ARROW will handle all those details transparently without the user needing to understand the details.&lt;br /&gt;
export I_MPI_HYDRA_BOOTSTRAP=ssh&lt;br /&gt;
export MPI_DEVICE=rdma:ofa-v2-ib0&lt;br /&gt;
export UCX_NET_DEVICES=qib0:1&lt;br /&gt;
#&lt;br /&gt;
# it doesn&#039;t make much sense, but in this example we are executing the OS command &amp;quot;uptime&amp;quot; on all cores&lt;br /&gt;
# of the nodes allocated to this job. The output from each core is written to the file info.log. We&lt;br /&gt;
# will find 56 lines of output in the file info.log, each created by the corresponding core executing&lt;br /&gt;
# the uptime command.&lt;br /&gt;
mpirun uptime &amp;gt; info.log&lt;br /&gt;
#&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
A good queue to test scripts is the &#039;&#039;&#039;&amp;quot;xeon28&amp;quot;&#039;&#039;&#039; queue. In the queue, we have 2 14-core Xeon processers per node, so that means that each node has 56 actual cores. We do not consider hyperthreading when doing parallel computing. 56 actual cores is what&#039;s being used here. The job submission will look like this:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
qsub -q xeon28 -l walltime=1:0:0 -l select=2:ncpus=28:mpiprocs=28:mem=60G parallel.job&lt;br /&gt;
        ^                  ^               ^       ^           ^      ^ ^ ^&lt;br /&gt;
        |                  |               |       |           |      | | + --- the name of the job script to execute&lt;br /&gt;
        |                  |               |       |           |      | + ----- don&#039;t forget to specify gigabytes&lt;br /&gt;
        |                  |               |       |           |      + ------- the amount of memory to request per node&lt;br /&gt;
        |                  |               |       |           + -------------- the number of MPI tasks per nodes&lt;br /&gt;
        |                  |               |       + -------------------------- the number of cores per node&lt;br /&gt;
        |                  |               + ---------------------------------- the number of nodes to select in the queue&lt;br /&gt;
        |                   + ------------------------------------------------- the requested time, in this case 1h&lt;br /&gt;
        + --------------------------------------------------------------------- the queue to be used for the job&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
At this point, the job has created a file &amp;quot;info.log&amp;quot; with 56 lines, one per core:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
 22:26:05 up 23 days,  9:28,  0 users,  load average: 0.00, 0.00, 0.00&lt;br /&gt;
 22:26:05 up 23 days,  9:28,  0 users,  load average: 0.00, 0.00, 0.00&lt;br /&gt;
 22:26:05 up 23 days,  9:28,  0 users,  load average: 0.00, 0.00, 0.00&lt;br /&gt;
 22:26:05 up 23 days,  9:28,  0 users,  load average: 0.00, 0.00, 0.00&lt;br /&gt;
 22:26:05 up 23 days,  9:28,  0 users,  load average: 0.00, 0.00, 0.00&lt;br /&gt;
 22:26:05 up 23 days,  9:28,  0 users,  load average: 0.00, 0.00, 0.00&lt;br /&gt;
 22:26:05 up 23 days,  9:53,  0 users,  load average: 0.06, 0.03, 0.00&lt;br /&gt;
 22:26:05 up 23 days,  9:53,  0 users,  load average: 0.06, 0.03, 0.00&lt;br /&gt;
 22:26:05 up 23 days,  9:53,  0 users,  load average: 0.06, 0.03, 0.00&lt;br /&gt;
...&lt;br /&gt;
&amp;lt;/PRE&amp;gt; &lt;br /&gt;
&lt;br /&gt;
In this simple example, the lines look all the same. Upon close examination through, you can find slightly different values for some of the lines. Some lines say that the machine is up for 23 days and 9:28, while others say 23 days and 9:53. Because all 28 cores of a node would see the same uptime of the server, half of the entries show one time stamp, and the other 28 cores show the other one. That demonstrates that the 56 processes have been running independently on 2 nodes.&lt;br /&gt;
&lt;br /&gt;
===Embedded Job Resource Requests===&lt;br /&gt;
&lt;br /&gt;
The job script can be modified to embed the resource requests in form of a series of &#039;&#039;&#039;#PBS&#039;&#039;&#039; statements at the beginning of the script file. This is a very common practice use at many HPC installations and job submission engines. Let&#039;s go back to the previous example where we run the script on two nodes in parallel. That is the &#039;&#039;&#039;&amp;quot;parallel.job&amp;quot;&#039;&#039;&#039; script file again:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
#!/bin/bash&lt;br /&gt;
#&lt;br /&gt;
#PBS -q xeon28&lt;br /&gt;
#PBS -l walltime=1:0:0&lt;br /&gt;
#PBS -l select=2:ncpus=28:mpiprocs=28:mem=60G&lt;br /&gt;
#&lt;br /&gt;
# the following ensures that you will change into the directory where you are&lt;br /&gt;
# submitting the job from.&lt;br /&gt;
cd $PBS_O_WORKDIR&lt;br /&gt;
#&lt;br /&gt;
# to execute a simple command on all of the cores of all of the nodes allocated to the job,&lt;br /&gt;
# we need to make one of the MPI versions available. Let&#039;s use one of the most up-to-date&lt;br /&gt;
# MPI library available on the cluster&lt;br /&gt;
module load intel/2024.2.0/mpi/2021.13&lt;br /&gt;
#&lt;br /&gt;
# now we are apply a few settings that ensure that the MPI library will use the highest-performing&lt;br /&gt;
# Infiniband Interconnect, as well as a few options to tell MPI how to interface nodes with&lt;br /&gt;
# each other and which specific Infiniband adapter to use. This is complex and requires in-depth&lt;br /&gt;
# knowledge of the QLogic Infiniband adapters we are using. It is unlikely that you will ever have to&lt;br /&gt;
# deal with these options, because the &amp;quot;module load&amp;quot; command for the engineering applications we provide&lt;br /&gt;
# on ARROW will handle all those details transparently without the user needing to understand the details.&lt;br /&gt;
export I_MPI_HYDRA_BOOTSTRAP=ssh&lt;br /&gt;
export MPI_DEVICE=rdma:ofa-v2-ib0&lt;br /&gt;
export UCX_NET_DEVICES=qib0:1&lt;br /&gt;
#&lt;br /&gt;
# it doesn&#039;t make much sense, but in this example we are executing the OS command &amp;quot;uptime&amp;quot; on all cores&lt;br /&gt;
# of the nodes allocated to this job. The output from each core is written to the file info.log. We&lt;br /&gt;
# will find 56 lines of output in the file info.log, each created by the corresponding core executing&lt;br /&gt;
# the uptime command.&lt;br /&gt;
mpirun uptime &amp;gt; info.log&lt;br /&gt;
#&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
If the resource requests are embedded within the file, they don&#039;t have to be specified on the command line any longer (the command line overrides the embedded specifications though). This may be convenient, because all the user has to do for job submission is the following:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
qsub parallel.job&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
Here is an example with more resource specifications and job settings that affect the behavior of the job:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
#!/bin/bash&lt;br /&gt;
#&lt;br /&gt;
#PBS -q xeon28&lt;br /&gt;
#PBS -l walltime=1:0:0&lt;br /&gt;
#PBS -l select=2:ncpus=28:mpiprocs=28:mem=60G&lt;br /&gt;
#PBS -A Account&lt;br /&gt;
#PBS -j oe&lt;br /&gt;
#PBS -N JobName&lt;br /&gt;
#PBS -e log.error&lt;br /&gt;
#PBS -o log.output&lt;br /&gt;
#PBS -m bae&lt;br /&gt;
#&lt;br /&gt;
...&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
I leave this to you as an exercise to figure out what the various options mean and how to specify them. There are many more, all documented in the PBS PRO manual (see above). Most of them are not terribly relevant and can be omitted.&lt;br /&gt;
&lt;br /&gt;
===Interactive Jobs===&lt;br /&gt;
&lt;br /&gt;
On ARROW, we don&#039;t restrict queues to be used only in batch mode. While &#039;&#039;&#039;batch mode&#039;&#039;&#039; is efficient for lining up a lot of work to be executed one after the other, ARROW has been designed to &#039;&#039;&#039;allow efficient model and software development in interactive mode&#039;&#039;&#039;. We have always ensured to have more computers than minimally needed to make it possible to dedicate resources to developers as needed, even if that means wasted CPU cycles. At times, we may ask you to limit the number of interactive jobs so that a large batch workload can be processed efficiently. This happens from time to time, and we have our users coordinate this with each other.&lt;br /&gt;
&lt;br /&gt;
Let&#039;s assume that you are developing an MPI application, or you are working on a complex &#039;&#039;&#039;OpenFOAM&#039;&#039;&#039; model that requires to start parallel processes over and over again just to find a bug and then fix it quickly. To do that, you can &#039;&#039;&#039;request an interactive job&#039;&#039;&#039; by adding the &#039;&#039;&#039;&amp;quot;-I&amp;quot;&#039;&#039;&#039; option to the job submission command (this is an uppercase I). Let&#039;s go to the parallel multi-node example from above:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
qsub -I -q xeon28 -l walltime=1:0:0 -l select=2:ncpus=28:mpiprocs=28:mem=60G&lt;br /&gt;
      ^    ^                  ^               ^       ^           ^      ^ ^&lt;br /&gt;
      |    |                  |               |       |           |      | + --- don&#039;t forget to specify gigabytes&lt;br /&gt;
      |    |                  |               |       |           |      + ----- the amount of memory to request per node&lt;br /&gt;
      |    |                  |               |       |           + ------------ the number of MPI tasks per nodes&lt;br /&gt;
      |    |                  |               |       + ------------------------ the number of cores per node&lt;br /&gt;
      |    |                  |               + -------------------------------- the number of nodes to select in the queue&lt;br /&gt;
      |    |                   + ----------------------------------------------- the requested time, in this case 1h&lt;br /&gt;
      |    + ------------------------------------------------------------------- the queue to be used for the job&lt;br /&gt;
      + ------------------------------------------------------------------------ request an interactive job &amp;lt;&amp;lt;===&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
When running interactive jobs with the &#039;&#039;&#039;&amp;quot;-I&amp;quot;&#039;&#039;&#039; parameter, we don&#039;t specify av job script at the end of the submission command. The interactive job will instead start (once the nodes are available) in interactive mode, meaning that the terminal session changes over from being a series of commands executed on the login server to being a series of commands being executed on the first node of the group of nodes that are allocated to the job. At this point, you can change to the desired working directory, but what you do with the allocated resources is entirely up to you. You can load modules, including MPI libraries, and then issue the commands for your application interactively and see how they execute. If you start an &#039;&#039;&#039;&amp;quot;mpirun&amp;quot;&#039;&#039;&#039;, the cores on your allocated secondary node will work as expected. There is no difference to batch mode, other than you having the ability to execute lines of commands at will.&lt;br /&gt;
&lt;br /&gt;
===Interactive Jobs with X-Windows GUI Applications===&lt;br /&gt;
&lt;br /&gt;
Interactive use can go further than that. With some of our software applications, like &#039;&#039;&#039;StarCCM+&#039;&#039;&#039;, you can run an &#039;&#039;&#039;interactive GUI application&#039;&#039;&#039; where you control the computational work from within the applications&#039; GUI. Within the GUI, you can control execution of the numerical solver that runs on as many cores as you requested, while being able to reconfigure the case through the GUI as well. Furthermore, you can visualize developing results on the fly by creating complex plots and visualizations.&lt;br /&gt;
&lt;br /&gt;
All that is need is an option &#039;&#039;&#039;&amp;quot;-X&amp;quot;&#039;&#039;&#039; being used as part of the job submission, like this:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
qsub -X -I -q xeon28 -l walltime=1:0:0 -l select=2:ncpus=28:mpiprocs=28:mem=60G&lt;br /&gt;
      ^  ^    ^                  ^               ^       ^           ^      ^ ^&lt;br /&gt;
      |  |    |                  |               |       |           |      | + --- don&#039;t forget to specify gigabytes&lt;br /&gt;
      |  |    |                  |               |       |           |      + ----- the amount of memory to request per node&lt;br /&gt;
      |  |    |                  |               |       |           + ------------ the number of MPI tasks per nodes&lt;br /&gt;
      |  |    |                  |               |       + ------------------------ the number of cores per node&lt;br /&gt;
      |  |    |                  |               + -------------------------------- the number of nodes to select in the queue&lt;br /&gt;
      |  |    |                   + ----------------------------------------------- the requested time, in this case 1h&lt;br /&gt;
      |  |    + ------------------------------------------------------------------- the queue to be used for the job&lt;br /&gt;
      |  + ------------------------------------------------------------------------ request an interactive job&lt;br /&gt;
      + --------------------------------------------------------------------------- request GUI capabilities &amp;lt;&amp;lt;===&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
===Running Multiple Jobs on Single Nodes===&lt;br /&gt;
&lt;br /&gt;
A feature that is new on ARROW is the ability to run multiple jobs on a single node. Let&#039;s assume that you are performing a sensitivity analysis on an existing model, and the model is simple enough to return results within a reasonable time on just a few cores of a higher end machine (maybe you are running SMP versions of &#039;&#039;&#039;LS-Dyna&#039;&#039;&#039;). Our high end machines have 64 cores, so lets assume you have an &#039;&#039;&#039;LS-Dyna&#039;&#039;&#039; model that runs well on 8 cores and doesn&#039;t use a whole lot of memory. In this case, you can submit individual jobs that request simply 8 cores and a fraction of the available memory available on the node, and all jobs execute independently from each other. Each job is fit into a slot where available. It is not very different from using whole nodes for everything. The important consideration is that each job is cleanly constrained into it&#039;s allotted resources using the &#039;&#039;&#039;CGROUPS&#039;&#039;&#039; functionality of modern operating systems. Because an abusive user cannot use more cores or more memory than allocated to his job, other users can safely run smaller jobs on the same node.&lt;br /&gt;
&lt;br /&gt;
Lets assume that we have a number of smaller jobs that we want to run on a single node in the &#039;&#039;&#039;&amp;quot;xeon28&amp;quot;&#039;&#039;&#039; queue. Each job would be submitted by using reduced resources that allow for sharing but that  guarantee that the jobs will be run successfully. In this case, you can &#039;&#039;&#039;submit many jobs&#039;&#039;&#039; in the following manner (with a job script for the small jobs, each of which can &#039;&#039;&#039;request varying resources&#039;&#039;&#039; if needed - some may want to run on 5 cores, others on 3):&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
#!/bin/bash&lt;br /&gt;
#&lt;br /&gt;
# the following ensures that you will change into the directory where you are&lt;br /&gt;
# submitting the job from.&lt;br /&gt;
cd $PBS_O_WORKDIR&lt;br /&gt;
#&lt;br /&gt;
# now we sleep for 300 seconds and waste time. This is a placeholder for your application,&lt;br /&gt;
# which would be doing useful work for you.&lt;br /&gt;
sleep 300&lt;br /&gt;
#&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
Now we submit a variety of these jobs (11 total in this example) to the &#039;&#039;&#039;&amp;quot;xeon28&amp;quot;&#039;&#039;&#039; queue for execution (note that the first few jobs request different amounts of memory and core counts):&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
qsub -q xeon28 -l walltime=1:0:0 -l select=1:ncpus=12:mpiprocs=12:mem=5G small.job &lt;br /&gt;
qsub -q xeon28 -l walltime=1:0:0 -l select=1:ncpus=10:mpiprocs=10:mem=7G small.job &lt;br /&gt;
qsub -q xeon28 -l walltime=1:0:0 -l select=1:ncpus=8:mpiprocs=8:mem=9G small.job &lt;br /&gt;
qsub -q xeon28 -l walltime=1:0:0 -l select=1:ncpus=16:mpiprocs=16:mem=20G small.job &lt;br /&gt;
qsub -q xeon28 -l walltime=1:0:0 -l select=1:ncpus=2:mpiprocs=2:mem=2G small.job &lt;br /&gt;
qsub -q xeon28 -l walltime=1:0:0 -l select=1:ncpus=2:mpiprocs=2:mem=2G small.job &lt;br /&gt;
qsub -q xeon28 -l walltime=1:0:0 -l select=1:ncpus=2:mpiprocs=2:mem=2G small.job &lt;br /&gt;
qsub -q xeon28 -l walltime=1:0:0 -l select=1:ncpus=2:mpiprocs=2:mem=2G small.job &lt;br /&gt;
qsub -q xeon28 -l walltime=1:0:0 -l select=1:ncpus=2:mpiprocs=2:mem=2G small.job &lt;br /&gt;
qsub -q xeon28 -l walltime=1:0:0 -l select=1:ncpus=2:mpiprocs=2:mem=2G small.job &lt;br /&gt;
qsub -q xeon28 -l walltime=1:0:0 -l select=1:ncpus=2:mpiprocs=2:mem=2G small.job &lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
They are now running in the order of submission, allocated on as few nodes in the &amp;quot;xeon28&amp;quot; queue as necessary. Only 2 nodes are being loaded quite heavily, and 4 more cores are in use on a third node.&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
Nov 20 23:34 ley@login3:myjobfolder$ qstat -an1&lt;br /&gt;
&lt;br /&gt;
pbs: &lt;br /&gt;
                                                            Req&#039;d  Req&#039;d   Elap&lt;br /&gt;
Job ID          Username Queue    Jobname    SessID NDS TSK Memory Time  S Time&lt;br /&gt;
--------------- -------- -------- ---------- ------ --- --- ------ ----- - -----&lt;br /&gt;
&lt;br /&gt;
3082.pbs        ley      xeon28   small.job  813221   1  12    5gb 01:00 R 00:01 p001/0*12&lt;br /&gt;
3083.pbs        ley      xeon28   small.job  813288   1  10    7gb 01:00 R 00:01 p001/1*10&lt;br /&gt;
3084.pbs        ley      xeon28   small.job  671792   1   8    9gb 01:00 R 00:01 p002/0*8&lt;br /&gt;
3085.pbs        ley      xeon28   small.job  671845   1  16   20gb 01:00 R 00:01 p002/1*16&lt;br /&gt;
3086.pbs        ley      xeon28   small.job  813361   1   2    2gb 01:00 R 00:00 p001/2*2&lt;br /&gt;
3087.pbs        ley      xeon28   small.job  813413   1   2    2gb 01:00 R 00:00 p001/3*2&lt;br /&gt;
3088.pbs        ley      xeon28   small.job  813464   1   2    2gb 01:00 R 00:00 p001/4*2&lt;br /&gt;
3089.pbs        ley      xeon28   small.job  671912   1   2    2gb 01:00 R 00:00 p002/2*2&lt;br /&gt;
3090.pbs        ley      xeon28   small.job  671969   1   2    2gb 01:00 R 00:00 p002/3*2&lt;br /&gt;
3091.pbs        ley      xeon28   small.job  632092   1   2    2gb 01:00 R 00:00 p003/0*2&lt;br /&gt;
3092.pbs        ley      xeon28   small.job  632100   1   2    2gb 01:00 R 00:00 p003/1*2&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
This is a particularly effective strategy to run concurrently many cases that don&#039;t scale well beyond a few cores. When running them on fewer cores but many of them at the same time, the overall processing rate will be much higher than executing them the traditional way.&lt;br /&gt;
&lt;br /&gt;
===Running Jobs using GPUs===&lt;br /&gt;
&lt;br /&gt;
The principle of running multiple jobs on a single node becomes particularly important when using servers equipped with &#039;&#039;&#039;GPUs&#039;&#039;&#039; for &#039;&#039;&#039;ML/AI&#039;&#039;&#039; applications. The cluster doesn&#039;t have a whole lot of &#039;&#039;&#039;GPUs&#039;&#039;&#039; at this point. We have three machines with three &#039;&#039;&#039;A4000&#039;&#039;&#039; GOUs, a &#039;&#039;&#039;total of 9 A4000 GPUs&#039;&#039;&#039;. Then we have a much more powerful single machine with our &#039;&#039;&#039;four A6000 GPUs&#039;&#039;&#039;.&lt;br /&gt;
&lt;br /&gt;
Using multiple GPUs in a single application is still something where the software has to be designed with hardware in mind. GPUs have several methods of communicating with each other, e.g. very fast &#039;&#039;&#039;NVLINK&#039;&#039;&#039; between pairs of GPUs, GPUs being directly connected to one of the two CPUs in the system and thus being able to communicate faster, and GPUs that have to jump between processors when communicating, and then the whole issue of having to go possibly through PCIe bridges.&lt;br /&gt;
&lt;br /&gt;
On our system, we are providing the ability to &#039;&#039;&#039;work mostly with individual GPUs&#039;&#039;&#039;. Users can also reserve entire nodes and develop or run applications that are adapted to that hardware, including several GPUs installed on that node. One thing we do not provide is the ability of GPU to GPU communication between nodes. Thus, a job cannot request more than one GPU node at a time.&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
qsub -q a4000 -I -l walltime=1:0:0 -l select=1:ncpus=5:mem=150G:ngpus=1&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
With these specifications, three single GPU jobs can run on a single server. Each job sees only one of the reserved GPU.&lt;br /&gt;
&lt;br /&gt;
To run a massive GPU job on 64 cores with 4 &#039;&#039;&#039;A6000 GPUs&#039;&#039;&#039;, submit the job like this:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
qsub -q a6000 -I -l walltime=1:0:0 -l select=1:ncpus=64:mem=725G:ngpus=4&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
===Manual pages for qsub===&lt;br /&gt;
&lt;br /&gt;
To learn more about the &amp;quot;qsub&amp;quot; command, you can use the command &amp;quot;man qsub&amp;quot;, which will print a lot of detailed information about the capabilities of this command.&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
$ man qsub&lt;br /&gt;
&lt;br /&gt;
qsub(1B)                                               PBS Professional                                              qsub(1B)&lt;br /&gt;
&lt;br /&gt;
NAME&lt;br /&gt;
       qsub - submit a job to PBS&lt;br /&gt;
&lt;br /&gt;
SYNOPSIS&lt;br /&gt;
       qsub [-a &amp;lt;date and time&amp;gt;] [-A &amp;lt;account string&amp;gt;] [-c &amp;lt;checkpoint spec&amp;gt;]&lt;br /&gt;
            [-C &amp;lt;directive prefix&amp;gt;] [-e &amp;lt;path&amp;gt;] [-f] [-h]&lt;br /&gt;
            [-I [-G [-- &amp;lt;GUI application/script&amp;gt;]] | [-X]] [-j &amp;lt;join&amp;gt;]&lt;br /&gt;
            [-J &amp;lt;range&amp;gt; [%&amp;lt;max subjobs]] [-k &amp;lt;discard&amp;gt;] [-l &amp;lt;resource list&amp;gt;]&lt;br /&gt;
            [-m &amp;lt;mail events&amp;gt;] [-M &amp;lt;user list&amp;gt;] [-N &amp;lt;name&amp;gt;] [-o &amp;lt;path&amp;gt;]&lt;br /&gt;
            [-p &amp;lt;priority&amp;gt;] [-P &amp;lt;project&amp;gt;] [-q &amp;lt;destination&amp;gt;] [-r &amp;lt;y|n&amp;gt;]&lt;br /&gt;
            [-R &amp;lt;remove options&amp;gt;] [-S &amp;lt;path list&amp;gt;] [-u &amp;lt;user list&amp;gt;]&lt;br /&gt;
            [-v &amp;lt;variable list&amp;gt;] [-V] [-W &amp;lt;additional attributes&amp;gt;] [-z]&lt;br /&gt;
            [- | &amp;lt;script&amp;gt; | -- &amp;lt;executable&amp;gt; [&amp;lt;arguments to executable&amp;gt;]]&lt;br /&gt;
       qsub --version&lt;br /&gt;
&lt;br /&gt;
DESCRIPTION&lt;br /&gt;
       You use the qsub command to submit a batch job to PBS.  Submitting a PBS job specifies a task, requests resources, and&lt;br /&gt;
       sets job attributes.&lt;br /&gt;
... &amp;lt;many more pages&amp;gt;&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
==LS-Dyna on the ARROW Cluster==&lt;br /&gt;
&lt;br /&gt;
===Currently Available LS-Dyna Versions===&lt;br /&gt;
&lt;br /&gt;
The following is a list of &#039;&#039;&#039;LS-Dyna versions&#039;&#039;&#039; available on &#039;&#039;&#039;ARROW&#039;&#039;&#039; after the latest reconfiguration of the system. As per LSTC/ANSYS, &#039;&#039;&#039;versions before 14.0.0 are not necessarily fully supported any longer&#039;&#039;&#039; because they are supposedly not compatible with modern operating systems and cannot be made to work reliably. We have tested the listed older versions of LS-Dyna and they have passed basic tests. They may not behave exactly as they did on the old CentOS 7 operating system, and time will show whether they can still be used or whether you will need to update your models and use a fully supported version.&lt;br /&gt;
&lt;br /&gt;
All versions are loaded using the &#039;&#039;&#039;&amp;quot;module load&amp;quot;&#039;&#039;&#039; command. Versions can be listed with the &#039;&#039;&#039;&amp;quot;module avail ls-dyna&amp;quot;&#039;&#039;&#039; command. To load one of the modules, use the following syntax:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
module load ls-dyna/14.2.0/mpi-d8-ifort190-avx512&lt;br /&gt;
            ^       ^      ^   ^  ^        ^&lt;br /&gt;
            |       |      |   |  |        + --- specify the extended instruction set needed for execution&lt;br /&gt;
            |       |      |   |  + ------------ load the version of the compiler that was used to create this &lt;br /&gt;
            |       |      |   + --------------- load the version that supports double precision variables&lt;br /&gt;
            |       |      + ------------------- load the MPP (MPI) version of LS-Dyna&lt;br /&gt;
            |       + -------------------------- load specifically version 14.2.0&lt;br /&gt;
            + ---------------------------------- load a version of LS-Dyna&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
The version string is composed of multiple elements to indicate variants in compilers and compiler options. Use the following guideline to choose an appropriate version to load:&lt;br /&gt;
&lt;br /&gt;
* &#039;&#039;&#039;&amp;quot;1&amp;quot;&#039;&#039;&#039; or &#039;&#039;&#039;&amp;quot;mpi&amp;quot;&#039;&#039;&#039; indicates whether this is a single node version of LS-Dyna (&#039;&#039;&#039;SMP&#039;&#039;&#039;) or whether this is a multi-node MPI version (&#039;&#039;&#039;MPP&#039;&#039;&#039;). All MPI versions use the &#039;&#039;&#039;IntelMPI 2022&#039;&#039;&#039; libraries which have been tested thoroughly on ARROW. MPI versions will use the Infiniband Network of ARROW for high-speed and low-latency inter-process communication using RDMA (remote direct memory access).&lt;br /&gt;
* All LS-Dyna versions are available in either &#039;&#039;&#039;floating point&#039;&#039;&#039; or &#039;&#039;&#039;double precision variants&#039;&#039;&#039;. Floating point variants use 4 bytes to represent a value, and double precision variants use 8 bytes. There are pros and cons for choosing one over the other variant. With regards to computational efficiency, both perform nearly the same because all machines are equipped with 64-bit CPUs.&lt;br /&gt;
** &#039;&#039;&#039;&amp;quot;f4&amp;quot;&#039;&#039;&#039; floating point versions&lt;br /&gt;
*** Pros: These require significantly less memory to run. Results occupy less disk space, and can be transferred significantly faster into and out of ARROW.&lt;br /&gt;
*** Cons: The numerical resolution is limited to 7 significant digits, which is often undesirable when dealing with mathematical operations on small and large numbers at the same time.&lt;br /&gt;
** &#039;&#039;&#039;&amp;quot;r8&amp;quot;&#039;&#039;&#039; double precision versions&lt;br /&gt;
*** Pros: The numerical resolution is about twice the number of significant digits compare to &amp;quot;f4&amp;quot;, which helps when when dealing with mathematical operations on small and large numbers at the same time.&lt;br /&gt;
*** Cons: These require more memory to run. Results occupy more disk space, and it takes longer to transfer data into and out of ARROW.&lt;br /&gt;
* There are two more identifiers to choose from when it comes to the variants of the executables: the specific compiler used to create the executable and the specific processor instruction set required for running the executable.&lt;br /&gt;
** For modern versions of LS-Dyna, two compilers have been used by the developers to create LS-Dyna executables: the &#039;&#039;&#039;Intel Fortran Compiler&#039;&#039;&#039; and the &#039;&#039;&#039;AOCC (AMD Optimizing C/C++ and Fortran) compiler&#039;&#039;&#039;. Both variants of the software are supported on ARROW. This gives users the opportunity to choose an alternate variant of the same LS-Dyna version when running into bugs or crashes.&lt;br /&gt;
** The variants based on the various instruction set extensions (&#039;&#039;&#039;SSE2&#039;&#039;&#039;, &#039;&#039;&#039;AVX2&#039;&#039;&#039;, &#039;&#039;&#039;AVX512&#039;&#039;&#039;, and so on) gives users even more options when choosing an alternate LS-Dyna variant of the same version when running into bugs or crashes. These instruction sets are mostly related to performance gains on specific processors. We have not performed thorough performance tests and cannot recommend specific versions right now.&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
$ module avail ls-dyna&lt;br /&gt;
--------------------------------------------- /shared/apps/modulefiles ---------------------------------------------&lt;br /&gt;
ls-dyna/09.3.1/1-d8-ifort131           ls-dyna/12.2.1/mpi-f4-ifort160-sse2    ls-dyna/14.2.0/mpi-f4-ifort190-avx512  &lt;br /&gt;
ls-dyna/09.3.1/1-f4-ifort131           ls-dyna/12.2.2/1-d8-ifort160-sse2      ls-dyna/14.2.0/mpi-f4-ifort190-sse2    &lt;br /&gt;
ls-dyna/09.3.1/mpi-d8-ifort131-avx2    ls-dyna/12.2.2/1-f4-ifort160-sse2      ls-dyna/15.0.2/1-d8-ifort190-sse2      &lt;br /&gt;
ls-dyna/09.3.1/mpi-d8-ifort131-avx512  ls-dyna/12.2.2/mpi-d8-aocc400-avx2     ls-dyna/15.0.2/1-f4-ifort190-sse2      &lt;br /&gt;
ls-dyna/09.3.1/mpi-f4-ifort131-avx2    ls-dyna/12.2.2/mpi-d8-ifort160-avx2    ls-dyna/15.0.2/mpi-d8-aocc400-avx2     &lt;br /&gt;
ls-dyna/09.3.1/mpi-f4-ifort131-avx512  ls-dyna/12.2.2/mpi-d8-ifort160-sse2    ls-dyna/15.0.2/mpi-d8-ifort190-avx2    &lt;br /&gt;
ls-dyna/10.2.0/1-d8-ifort160           ls-dyna/12.2.2/mpi-f4-aocc400-avx2     ls-dyna/15.0.2/mpi-d8-ifort190-avx512  &lt;br /&gt;
ls-dyna/10.2.0/1-f4-ifort160           ls-dyna/12.2.2/mpi-f4-ifort160-avx2    ls-dyna/15.0.2/mpi-d8-ifort190-sse2    &lt;br /&gt;
ls-dyna/11.0.0/1-d8-ifort160           ls-dyna/12.2.2/mpi-f4-ifort160-sse2    ls-dyna/15.0.2/mpi-f4-aocc400-avx2     &lt;br /&gt;
ls-dyna/11.0.0/1-f4-ifort160           ls-dyna/13.0.0/1-d8-ifort190           ls-dyna/15.0.2/mpi-f4-ifort190-avx2    &lt;br /&gt;
ls-dyna/11.1.0/1-d8-ifort160-sse2      ls-dyna/13.0.0/1-f4-ifort190           ls-dyna/15.0.2/mpi-f4-ifort190-avx512  &lt;br /&gt;
ls-dyna/11.1.0/1-f4-ifort160-sse2      ls-dyna/13.0.0/mpi-d8-ifort190-avx2    ls-dyna/15.0.2/mpi-f4-ifort190-sse2    &lt;br /&gt;
ls-dyna/11.2.0/1-d8-ifort160           ls-dyna/13.0.0/mpi-d8-ifort190-sse2    ls-dyna/16.0.0/1-d8-aocc420-avx2       &lt;br /&gt;
ls-dyna/11.2.0/1-f4-ifort160           ls-dyna/13.0.0/mpi-f4-ifort190-avx2    ls-dyna/16.0.0/1-d8-aocc420-avx512     &lt;br /&gt;
ls-dyna/11.2.0/mpi-f4-ifort160-avx2    ls-dyna/13.0.0/mpi-f4-ifort190-sse2    ls-dyna/16.0.0/1-d8-ifort190-sse2      &lt;br /&gt;
ls-dyna/11.2.0/mpi-f4-ifort160-sse2    ls-dyna/13.1.0/mpi-d8-aocc310-avx2     ls-dyna/16.0.0/1-f4-aocc420-avx2       &lt;br /&gt;
ls-dyna/11.2.1/1-d8-ifort160           ls-dyna/13.1.0/mpi-d8-ifort190-avx2    ls-dyna/16.0.0/1-f4-aocc420-avx512     &lt;br /&gt;
ls-dyna/11.2.1/1-f4-ifort160           ls-dyna/13.1.0/mpi-d8-ifort190-sse2    ls-dyna/16.0.0/1-f4-ifort190-sse2      &lt;br /&gt;
ls-dyna/11.2.1/mpi-d8-ifort160-avx2    ls-dyna/13.1.0/mpi-f4-aocc310-avx2     ls-dyna/16.0.0/mpi-d8-aocc420-avx2     &lt;br /&gt;
ls-dyna/11.2.1/mpi-d8-ifort160-sse2    ls-dyna/13.1.0/mpi-f4-ifort190-avx2    ls-dyna/16.0.0/mpi-d8-aocc420-avx512   &lt;br /&gt;
ls-dyna/11.2.1/mpi-f4-ifort160-avx2    ls-dyna/13.1.0/mpi-f4-ifort190-sse2    ls-dyna/16.0.0/mpi-d8-ifort190-avx2    &lt;br /&gt;
ls-dyna/11.2.1/mpi-f4-ifort160-sse2    ls-dyna/13.1.1/mpi-d8-ifort190-avx2    ls-dyna/16.0.0/mpi-d8-ifort190-avx512  &lt;br /&gt;
ls-dyna/11.2.2/mpi-d8-ifort160-avx2    ls-dyna/13.1.1/mpi-d8-ifort190-sse2    ls-dyna/16.0.0/mpi-d8-ifort190-sse2    &lt;br /&gt;
ls-dyna/11.2.2/mpi-d8-ifort160-sse2    ls-dyna/13.1.1/mpi-f4-ifort190-avx2    ls-dyna/16.0.0/mpi-f4-aocc420-avx2     &lt;br /&gt;
ls-dyna/11.2.2/mpi-f4-ifort160-avx2    ls-dyna/13.1.1/mpi-f4-ifort190-sse2    ls-dyna/16.0.0/mpi-f4-aocc420-avx512   &lt;br /&gt;
ls-dyna/11.2.2/mpi-f4-ifort160-sse2    ls-dyna/14.0.0/1-d8-ifort190           ls-dyna/16.0.0/mpi-f4-ifort190-avx2    &lt;br /&gt;
ls-dyna/12.1.0/1-d8-ifort160           ls-dyna/14.0.0/1-f4-ifort190           ls-dyna/16.0.0/mpi-f4-ifort190-avx512  &lt;br /&gt;
ls-dyna/12.1.0/1-f4-aocc310            ls-dyna/14.0.0/mpi-d8-aocc310-avx2     ls-dyna/16.0.0/mpi-f4-ifort190-sse2    &lt;br /&gt;
ls-dyna/12.1.0/1-f4-ifort160           ls-dyna/14.0.0/mpi-d8-ifort190-avx2    ls-dyna/16.1.0/mpi-d8-aocc420-avx2     &lt;br /&gt;
ls-dyna/12.1.0/mpi-d8-aocc310-avx2     ls-dyna/14.0.0/mpi-d8-ifort190-sse2    ls-dyna/16.1.0/mpi-d8-aocc420-avx512   &lt;br /&gt;
ls-dyna/12.1.0/mpi-d8-ifort160-avx2    ls-dyna/14.0.0/mpi-f4-ifort190-avx2    ls-dyna/16.1.0/mpi-d8-ifort190-avx2    &lt;br /&gt;
ls-dyna/12.1.0/mpi-d8-ifort160-sse2    ls-dyna/14.0.0/mpi-f4-ifort190-sse2    ls-dyna/16.1.0/mpi-d8-ifort190-avx512  &lt;br /&gt;
ls-dyna/12.1.0/mpi-f4-aocc310-avx2     ls-dyna/14.1.0/1-d8-ifort190-sse2      ls-dyna/16.1.0/mpi-d8-ifort190-sse2    &lt;br /&gt;
ls-dyna/12.1.0/mpi-f4-ifort160-avx2    ls-dyna/14.1.0/1-f4-ifort190-sse2      ls-dyna/16.1.0/mpi-f4-aocc420-avx2     &lt;br /&gt;
ls-dyna/12.1.0/mpi-f4-ifort160-sse2    ls-dyna/14.1.0/mpi-d8-aocc400-avx2     ls-dyna/16.1.0/mpi-f4-aocc420-avx512&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
===Submitting an LS-Dyna Job===&lt;br /&gt;
&lt;br /&gt;
&amp;lt;BLOCKQUOTE&amp;gt;&lt;br /&gt;
[[File:Attention.jpg|25px]] &#039;&#039;&#039;IMPORTANT NOTE:&#039;&#039;&#039; The job/queue manager can now track the number of LS-Dyna licenses given out to individual&lt;br /&gt;
jobs. At submission time, it is not possible to know what software a user may run. But by adding the clause &amp;quot;-l dynalic&amp;quot; at submission time,&lt;br /&gt;
the queue manager can calculate the total number of cores required and keep track of LS-Dyna licenses used by the job. When loading a version of LS-Dyna, a check will be performed, and LS-Dyna will be prevented from running if the &amp;quot;-l dynalic&amp;quot; clause was not used when submitting the job.&#039;&#039;&lt;br /&gt;
&lt;br /&gt;
&amp;lt;!--&lt;br /&gt;
&#039;&#039;The job/queue manager can track the number of LS-Dyna licenses to some degree. If all &#039;&#039;&#039;LS-Dyna users&#039;&#039;&#039; cooperate and use a script like the one shown below when submitting their jobs, the total number of concurrent &#039;&#039;&#039;LS-Dyna licenses&#039;&#039;&#039; will be tracked by the job manager correctly. That means that users can submit any number of LS-Dyna jobs, and jobs will only start when a sufficient number of licenses is available. This is managed by the &#039;&#039;&#039;&amp;quot;dynalic&amp;quot;&#039;&#039;&#039; resource at the end of the select statement. In this example, a 2-node job on 64-core nodes will need a total of &#039;&#039;&#039;&amp;quot;dynalic=128&amp;quot;&#039;&#039;&#039; licenses. This accounting breaks down when users don&#039;t use the &#039;&#039;&#039;&amp;quot;dynalic=XXX&amp;quot;&#039;&#039;&#039; statement, or when they don&#039;t calculate the number of licenses correctly. In that case, LS-Dyna jobs of all users are subject to sudden failure when LS-Dyna licenses run out. Please understand the importance of this specific setting in your job.&#039;&#039;&lt;br /&gt;
--&amp;gt;&lt;br /&gt;
&amp;lt;/BLOCKQUOTE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
Furthermore, careful consideration should be given with regards to choice of resources for an &#039;&#039;&#039;LS-Dyna job&#039;&#039;&#039;. With 64 cores available on a single node in the &#039;&#039;&#039;&amp;quot;epyc1&amp;quot;&#039;&#039;&#039; and &#039;&#039;&#039;&amp;quot;epyc2&amp;quot;&#039;&#039;&#039; queues, it may be counterproductive to run a job on two nodes instead of a single node. Users should run their jobs with different numbers of nodes and determine whether performance increases. It may well decrease when running a job on two or more nodes. The outcome of such tests will tell what the best allocation of resources will be.&lt;br /&gt;
&lt;br /&gt;
Most users use a job script like the following. All methods for job submission the the previous chapters apply as well, so there is a lot of flexibility:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
#!/bin/bash&lt;br /&gt;
#&lt;br /&gt;
#PBS -q epyc1&lt;br /&gt;
#PBS -l walltime=12:0:0&lt;br /&gt;
#PBS -l select=2:ncpus=64:mpiprocs=64:mem=225G,dynalic&lt;br /&gt;
#PBS -N JobName&lt;br /&gt;
#PBS -e log.error&lt;br /&gt;
#PBS -o log.output&lt;br /&gt;
#&lt;br /&gt;
cd $PBS_O_WORKDIR&lt;br /&gt;
#&lt;br /&gt;
module load ls-dyna/12.2.1/mpi-f4-ifort160-avx2&lt;br /&gt;
module load dynamore/current&lt;br /&gt;
#&lt;br /&gt;
mpirun ls-dyna i=main.k memory1=300m memory2=100m &amp;gt; dyna.log&lt;br /&gt;
#&lt;br /&gt;
# when using the Dynamore tools, you can start something like this at the end&lt;br /&gt;
DM.plotcprs.lnx -merge &amp;gt;&amp;gt; dyna.log&lt;br /&gt;
#&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
===LSTC Tools: LS-OPT and LS-PREPOST===&lt;br /&gt;
&lt;br /&gt;
For the new Rocky 9 cluster, I have not looked deeply into the ls-opt and ls-prepost packages that were installed. I noticed though that the LSTC server provided access to much newer versions of both software packages. If you would like to learn more or have a specific version in mind, I can easily download and install it for you.&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
$ module avail ls-opt&lt;br /&gt;
----------------------------------------------- /shared/apps/modulefiles ------------------------------------------------&lt;br /&gt;
ls-opt/5.1.1  ls-opt/6.0.0  ls-opt/7.0.0  ls-opt/7.0.2   ls-opt/2022R2  &lt;br /&gt;
ls-opt/5.2.1  ls-opt/6.1.0  ls-opt/7.0.1  ls-opt/2022R1  ls-opt/2023R1  &lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
To start the software, type:&lt;br /&gt;
 lsopt&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
$ module avail ls-prepost&lt;br /&gt;
----------------------------------------------- /shared/apps/modulefiles ------------------------------------------------&lt;br /&gt;
ls-prepost/4.5.10  ls-prepost/4.8.13  ls-prepost/4.8.30  ls-prepost/4.9.16  ls-prepost/4.10.7 &lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
To start the software, type:&lt;br /&gt;
 lsprepost&lt;br /&gt;
&lt;br /&gt;
===Dynamore Software===&lt;br /&gt;
&lt;br /&gt;
The Dynamore tools are available as a module:&lt;br /&gt;
&lt;br /&gt;
 module load dynamore/current&lt;br /&gt;
&lt;br /&gt;
We typically acquire a yearly license for the tools as we purchase licenses for LS-Dyna.&lt;br /&gt;
&lt;br /&gt;
===Vendor License File Installation===&lt;br /&gt;
&lt;br /&gt;
If you would like for us to install a vendor license for LS-Dyna models, please contact us for the required information. We can send you the general LS-Dyna license file to show the host ids for the license server. Using that information, your vendor should be able to create a vendor license file. Please send that file to us per Email or by other means.&lt;br /&gt;
&lt;br /&gt;
==StarCCM+ on the ARROW Cluster==&lt;br /&gt;
&lt;br /&gt;
===Currently Available StarCCM+ Versions===&lt;br /&gt;
&lt;br /&gt;
As of late 2025, we have the following versions of &#039;&#039;&#039;StarCCM+&#039;&#039;&#039; available on the cluster:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
$ module avail starccm&lt;br /&gt;
---------------------------- /shared/apps/modulefiles ----------------------------&lt;br /&gt;
starccm/15.02.007-R8  starccm/16.06.008-R8  starccm/18.06.006-R8  &lt;br /&gt;
starccm/15.02.009-R8  starccm/17.02.007-R8  starccm/19.02.009-R8  &lt;br /&gt;
starccm/15.04.008-R8  starccm/17.02.008-R8  starccm/20.04.007-R8  &lt;br /&gt;
starccm/15.06.008-R8  starccm/17.04.007-R8  starccm/20.06.007-R8  &lt;br /&gt;
starccm/16.02.008-R8  starccm/17.06.007-R8  &lt;br /&gt;
starccm/16.04.007-R8  starccm/18.04.008-R8&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
If using a &#039;&#039;&#039;single node&#039;&#039;&#039; for StarCCM+, job submission (for an interactive job) is simple and will use appropriate default settings:&lt;br /&gt;
&lt;br /&gt;
 qsub -I -X -q epyc1 -l walltime=20:00:00&lt;br /&gt;
&lt;br /&gt;
StarCCM+ can make use of the job scheduler attributes by automatically obtaining the number of cores and other resources from OpenPBS. In this case, the default number of cores and mpi processes for StarCCM+ are both 64 when using the epyc1 queue. So you can start your StarCCM+ run with:&lt;br /&gt;
&lt;br /&gt;
 module load starccm/15.02.007-R8 (or any other version)&lt;br /&gt;
 starccm+ -bs pbs&lt;br /&gt;
&lt;br /&gt;
In this case, there is no need for StarCCM+ to be told to run the case in parallel with the selected number of cores/mpiprocs.&lt;br /&gt;
&lt;br /&gt;
This can get a bit more complex when running on multiple nodes or when requesting high memory nodes. In that case you would use job submission parameters as shown below:&lt;br /&gt;
&lt;br /&gt;
 qsub -I -X -q epyc1 -l walltime=20:00:00,select=2:ncpus=64:mpiprocs=61:mem=500GB&lt;br /&gt;
&lt;br /&gt;
Requesting nodes that can satisfy those resources, two nodes with these attributes must exist. We have multiple nodes with 512GB in the epyc1 queue, meaning that this job will run on two machines that have at least the required amount of memory installed (on each node). The job will be queued until two machines like this will be available. If no machines with these resources exist, the job will stay in the queue forever. Therefore, you have to craft the submission string carefully.&lt;br /&gt;
&lt;br /&gt;
To accommodate high memory jobs, the nodes have been assigned priorities for assignment. Low memory jobs have the highest priority and will be assigned to nodes that can accommodate the request. High memory nodes have the lowest priority, meaning that they are the last ones given out to users. This makes it more likely that a high memory job can be run soon when the cluster is moderately loaded with jobs.&lt;br /&gt;
&lt;br /&gt;
StarCCM+ will always use the Intel MPI fabric. Other MPI versions do not work, even when selected on the command line.&lt;br /&gt;
&lt;br /&gt;
==OpenFOAM on the ARROW Cluster==&lt;br /&gt;
&lt;br /&gt;
===Currently Available OpenFOAM  Versions===&lt;br /&gt;
&lt;br /&gt;
As of late 2025, we have the following versions of OpenFOAM available on the cluster:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
$ module avail openfoam&lt;br /&gt;
------------ /shared/apps/modulefiles ------------&lt;br /&gt;
openfoam/9   openfoam/13      openfoam/v2312  &lt;br /&gt;
openfoam/10  openfoam/13-amd  openfoam/v2406  &lt;br /&gt;
openfoam/11  openfoam/v2212   &lt;br /&gt;
openfoam/12  openfoam/v2306&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
Contact us if you encounter problems; there can be various reasons why OpenFOAM may have trouble on certain hardware or when compiling dynamic code. When loading OpenFOAM modules, a number of dependencies will be automatically loaded for you, and you don&#039;t have to load those yourself. For example:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
$ module load openfoam/13&lt;br /&gt;
Loading openfoam/13&lt;br /&gt;
  Loading requirement: intel/2024.2.0/mpi/2021.13 gcc/gcc-12.1.0&lt;br /&gt;
&lt;br /&gt;
$ module list&lt;br /&gt;
Currently Loaded Modulefiles:&lt;br /&gt;
 1) intel/2024.2.0/mpi/2021.13   2) gcc/gcc-12.1.0   3) openfoam/13&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
In this case, OpenFOAM 13 loads the Intel 2024 MPI module, and loads the GCC compiler 12.1. OpenFOAM was compiled from source, and has been compiled specifically with that compiler and MPI version, so it make little sense to use other compilers or MPI libraries.&lt;br /&gt;
&lt;br /&gt;
Note: We have found a problem with running the Intel 2024 MPI library in the amd64 queue. Therefore, we have a modified module that uses the Intel 2022 library (I know -- Intel 2022 gives you the 2021 MPI libraries, but that is the way Intel distributes this software):&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
$ module load openfoam/13-amd &lt;br /&gt;
Loading mpi version 2021.7.0&lt;br /&gt;
Loading openfoam/13-amd&lt;br /&gt;
  Loading requirement: intel/2022.2.0/mpi/2021.7.0 gcc/gcc-12.1.0&lt;br /&gt;
&lt;br /&gt;
$ module list&lt;br /&gt;
Currently Loaded Modulefiles:&lt;br /&gt;
 1) intel/2022.2.0/mpi/2021.7.0   2) gcc/gcc-12.1.0   3) openfoam/13-amd&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
If you are compiling OpenFOAM yourself, the modules are of little help. You would need to select the appropriate MPI version and compiler before doing so, and then consistently load them before running your OpenFOAM executables. Within the &amp;quot;etc/bashrc&amp;quot; file in the source code tree, you want to set the MPI library to INTELMPI. As usual with self-compiled versions of OpenFOAM, you would &amp;quot;source etc/bashrc&amp;quot; to set up your personal environment to run your home-brew version of OpenFOAM. Contact us if you need to learn more about compiling OpenFOAM on the system.&lt;br /&gt;
&lt;br /&gt;
==Additional Software Applications and Libraries==&lt;br /&gt;
&lt;br /&gt;
===Loadable GCC Compiler Versions===&lt;br /&gt;
&lt;br /&gt;
The Rocky 9.6 operating system uses the GCC 11.5 compiler. That should be sufficient for most users when compiling your own applications. In case you need to use either a more up-to-date compiler, or if you need an older compiler for compatibility, we make the following versions available as loadable modules.&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
$ module avail gcc&lt;br /&gt;
------------ /shared/apps/modulefiles ------------&lt;br /&gt;
gcc/gcc-4.9.4  gcc/gcc-7.5.0  gcc/gcc-10.3.0  &lt;br /&gt;
gcc/gcc-5.5.0  gcc/gcc-8.5.0  gcc/gcc-11.3.0  &lt;br /&gt;
gcc/gcc-6.5.0  gcc/gcc-9.5.0  gcc/gcc-12.1.0&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
Additional versions can be created and made available as modules as well. If you need a specific version that is not currently available, please ask us to compiler and install it. If necessary, we may be able to provide access to other compilers, for example LLVM. We do not provide access to proprietary compilers at this time.&lt;br /&gt;
&lt;br /&gt;
===MPI Libraries and Runtimes===&lt;br /&gt;
&lt;br /&gt;
While we seem to have a variety of MPI versions and flavors available to users, the only MPI versions that allow us to run software over Infiniband are the Intel MPI libraries. Some of the installed alternatives are likely to fail, or will have a set of environment variables that have to be set. All major engineering software packages that we offer are pre-configured with specific MPI versions and settings that have been tested and/or provided by the vendors.&lt;br /&gt;
&lt;br /&gt;
Note: Some MPI libraries may seem to work. They may allow your MPI application to run. But inter-process network communication may travel through the rather slow and high-latency Ethernet fabric, making MPI applications very ineffective and are probably not worth while.&lt;br /&gt;
&lt;br /&gt;
===MatLab Runtimes===&lt;br /&gt;
&lt;br /&gt;
We can install MatLAB run time libraries as needed and have them available as loadable modules. Recently, we had a problem with MatLAB run time libraries being identified as security vulnerabilities. Contact us if you need them installed for one of your projects.&lt;br /&gt;
&lt;br /&gt;
===Anaconda and variants (miniconda etc)===&lt;br /&gt;
&lt;br /&gt;
Our current practice is to have users download and install their own versions of Anaconda and its variants in their own home directories. This allows for maximum flexibility when it comes to installable software modules, and users can maintain the installation, upgrades, and maintenance themselves. If you encounter issues, please contact us. One known side effect of Anaconda installations is a performance hit when starting your software, e.g. python scripts may take 30 seconds or more to execute. This is an artefact caused by the Lustre file system, which has been designed for large files accessible from many machines simultaneously. Performance on reading many small files has not been considered and is fairly poor. Again, contact us and we will design a solution for you as needed.&lt;/div&gt;</summary>
		<author><name>Ley</name></author>
	</entry>
	<entry>
		<id>https://wiki.anl.gov/wiki_tracc/index.php?title=Job_Submission_and_Monitoring&amp;diff=2489</id>
		<title>Job Submission and Monitoring</title>
		<link rel="alternate" type="text/html" href="https://wiki.anl.gov/wiki_tracc/index.php?title=Job_Submission_and_Monitoring&amp;diff=2489"/>
		<updated>2026-02-23T17:19:43Z</updated>

		<summary type="html">&lt;p&gt;Ley: /* Submitting an LS-Dyna Job */&lt;/p&gt;
&lt;hr /&gt;
&lt;div&gt;{| align=&amp;quot;right&amp;quot;&lt;br /&gt;
| __TOC__&lt;br /&gt;
|}&lt;br /&gt;
==Resource Summary View==&lt;br /&gt;
&lt;br /&gt;
To get started, users can query the overall status of resources on the cluster. The &#039;&#039;&#039;&amp;quot;qsum&amp;quot;&#039;&#039;&#039; script will list all queues and nodes, as well as how many are offline, down, free, or assigned to users. This is a script developed by our team, and may need to be updated if something goes wrong. Please contact us if you experience any problems.&lt;br /&gt;
&lt;br /&gt;
Each queue groups a number of nodes together based on their hardware and software configurations. Nodes can be part of more than one queue, and there are other complex details that we are ignoring here for the purpose of keeping it simple.&lt;br /&gt;
&lt;br /&gt;
===Queues===&lt;br /&gt;
&lt;br /&gt;
Here is a very brief summary of what each of the queues is, and how to use them efficiently:&lt;br /&gt;
&lt;br /&gt;
; a4000: This is a queue that has three 16-core CPU machines, each of which is furthermore equipped with three A4000 GPUs. That makes a total of 9 A4000 GPUs available to users. Neither the GPUs nor the processors are particularly powerful these days. The machines have 500GB of memory though, which makes for a good platform for experimenting with GPU capabilities.&lt;br /&gt;
; a6000: This is a queue that has only one 64-core CPU machines, and is equipped with four A6000 GPUs. The system can be upgraded to 8 A6000 GPUs if needed. This is a decent GPU machine that can take a solid workload these days. The machine has 750GB of memory, which makes for a good production platform.&lt;br /&gt;
; amd16: This is a queue with many of our older AMD-based 16-core machines, each of which has 30GB of memory. While individual machines are a bit outdated, they are all interconnected with Infiniband and can provide a solid production workload in multi-nodes jobs over MPI without blocking the more current (and thus expensive) systems.&lt;br /&gt;
; epyc1/epyc2: These are 2 separate queues with slightly different performance characteristics. Each of the groups is interconnected with Infiniband to provide a platform for large and demanding software packages, such as LS-Dyna and StarCCM+. They have between 250GB and 500GB of memory. Because licenses for these software packages are very expensive, they should use these two queues for making optimum use of limited core licenses available to each package.&lt;br /&gt;
; xeon28: This is a set of intermediate machines with 28 cores and 64GB of memory. They can be used for a variety of purposes, including MPI jobs and single node application software.&lt;br /&gt;
; virtual: This is a set of nodes without MPI capabilities. They are virtual machines with 32GB each. They can be used for higher demand applications that would interfere with the login nodes, and therefore with other users of these login machines. A user would submit interactive jobs to individual virtual machines and avoid any significant load on login nodes.&lt;br /&gt;
&lt;br /&gt;
===The Queue Summary Script (qsum)===&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
$ qsum&lt;br /&gt;
=============== a4000 ==========================================================&lt;br /&gt;
Queue: &amp;quot;a4000&amp;quot; / nodes: 3 / down: 0 / offline: 0 / busy: 0 / available: 3&lt;br /&gt;
      AVAILABLE (3): g001, g002, g003&lt;br /&gt;
=============== a6000 ==========================================================&lt;br /&gt;
Queue: &amp;quot;a6000&amp;quot; / nodes: 1 / down: 0 / offline: 0 / busy: 0 / available: 1&lt;br /&gt;
      AVAILABLE (1): lambda01&lt;br /&gt;
=============== amd16 ==========================================================&lt;br /&gt;
Queue: &amp;quot;amd16&amp;quot; / nodes: 33 / down: 2 / offline: 0 / busy: 2 / available: 29&lt;br /&gt;
           DOWN (2): n017, n030&lt;br /&gt;
            ley (2): n001, n002&lt;br /&gt;
     AVAILABLE (29): n003, n004, n005, n006, n007, n008, n009, n010, n011, n012&lt;br /&gt;
                     n013, n014, n015, n016, n018, n019, n020, n021, n022, n023&lt;br /&gt;
                     n024, n025, n026, n027, n028, n029, n031, n032, n039&lt;br /&gt;
=============== epyc1 ==========================================================&lt;br /&gt;
Queue: &amp;quot;epyc1&amp;quot; / nodes: 1 / down: 0 / offline: 0 / busy: 0 / available: 1&lt;br /&gt;
      AVAILABLE (1): a027&lt;br /&gt;
=============== epyc2 ==========================================================&lt;br /&gt;
Queue: &amp;quot;epyc2&amp;quot; / nodes: 20 / down: 0 / offline: 0 / busy: 5 / available: 15&lt;br /&gt;
            ley (2): a030, a031&lt;br /&gt;
         msitek (3): a028, a029, a032&lt;br /&gt;
     AVAILABLE (15): a033, a034, a035, a036, a037, a038, a039, a040, a041, a042&lt;br /&gt;
                     a043, a044, a045, a046, a047&lt;br /&gt;
=============== virtual ========================================================&lt;br /&gt;
Queue: &amp;quot;virtual&amp;quot; / nodes: 6 / down: 0 / offline: 0 / busy: 0 / available: 6&lt;br /&gt;
      AVAILABLE (6): v001, v002, v003, v004, v005, v006&lt;br /&gt;
=============== xeon28 =========================================================&lt;br /&gt;
Queue: &amp;quot;xeon28&amp;quot; / nodes: 12 / down: 0 / offline: 0 / busy: 0 / available: 12&lt;br /&gt;
     AVAILABLE (12): p001, p002, p003, p004, p005, p006, p007, p008, p009, p010&lt;br /&gt;
                     p011, p012&lt;br /&gt;
================================================================================&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
==Queue Status and Monitoring Jobs==&lt;br /&gt;
&lt;br /&gt;
===qstat===&lt;br /&gt;
&lt;br /&gt;
To find out out about the status of all running jobs on the cluster you can use the &#039;&#039;&#039;&amp;quot;qstat&amp;quot;&#039;&#039;&#039; command. Here is an example:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
$ qstat&lt;br /&gt;
&lt;br /&gt;
Nov 20 18:30 ley@login3:Plots$ qstat&lt;br /&gt;
Job id            Name             User              Time Use S Queue&lt;br /&gt;
----------------  ---------------- ----------------  -------- - -----&lt;br /&gt;
3023.pbs          STDIN            msitek            4144:14* R epyc2           &lt;br /&gt;
3029.pbs          STDIN            ley               76:46:53 R epyc2           &lt;br /&gt;
3032.pbs          STDIN            msitek            2879:52* R epyc2           &lt;br /&gt;
3033.pbs          STDIN            msitek            3687:29* R epyc2           &lt;br /&gt;
3048.pbs          foo.sh           james.cook               0 Q amd16           &lt;br /&gt;
3060.pbs          of13.sh          ley               310:47:* R epyc2           &lt;br /&gt;
3061.pbs          of13.sh          ley               308:37:* R epyc2           &lt;br /&gt;
3062.pbs          of13.sh          ley               308:02:* R epyc2           &lt;br /&gt;
3063.pbs          of13.sh          ley               308:15:* R epyc2&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
The first column shows the &#039;&#039;&#039;job id&#039;&#039;&#039;, a unique identifier for all jobs ever submitted to the cluster. This job id is important when killing jobs, or for other actions you may need to take.&lt;br /&gt;
&lt;br /&gt;
The next column shows the name of the job script. If the column shows &#039;&#039;&#039;STDIN&#039;&#039;&#039;, it means that this is an &#039;&#039;&#039;interactive job&#039;&#039;&#039; where a user can enter commands in a terminal window. This is particularly useful for model and software development task where the application has to be started and killed repeatedly.&lt;br /&gt;
&lt;br /&gt;
The owner of the job is shown next. These are the user names of the various people using the cluster.&lt;br /&gt;
&lt;br /&gt;
The last three columns indicate the current run time of the job, whether it is running (R) or waiting (Q) for execution. The last entry shows the queue in which the job is running.&lt;br /&gt;
&lt;br /&gt;
===qstat -an1===&lt;br /&gt;
&lt;br /&gt;
Adding a few options gives much more detail about each jobs.&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
qstat -an1&lt;br /&gt;
&lt;br /&gt;
Nov 20 13:09 ley@login3:Plots$ qstat -an1&lt;br /&gt;
&lt;br /&gt;
pbs: &lt;br /&gt;
                                                            Req&#039;d  Req&#039;d   Elap&lt;br /&gt;
Job ID          Username Queue    Jobname    SessID NDS TSK Memory Time  S Time&lt;br /&gt;
--------------- -------- -------- ---------- ------ --- --- ------ ----- - -----&lt;br /&gt;
3023.pbs        msitek   epyc2    STDIN      24360*   1  64  350gb 100:0 R 81:46 a028/0*64&lt;br /&gt;
3029.pbs        ley      epyc2    STDIN      21719*   2 128  100gb 200:0 R 72:31 a030/0*64+a031/0*64&lt;br /&gt;
3032.pbs        msitek   epyc2    STDIN      18102*   1  64  350gb 100:0 R 57:57 a029/0*64&lt;br /&gt;
3033.pbs        msitek   epyc2    STDIN      830486   1  64  350gb 100:0 R 57:53 a032/0*64&lt;br /&gt;
3048.pbs        james.c* amd16    foo.sh        --    1  28   30gb 01:00 Q   --   -- &lt;br /&gt;
3060.pbs        ley      epyc2    STDIN      763101   1  64  350gb 48:00 R 06:42 a033/0*64&lt;br /&gt;
3061.pbs        ley      epyc2    STDIN      763947   1  64  350gb 48:00 R 06:40 a034/0*64&lt;br /&gt;
3062.pbs        ley      epyc2    STDIN      761473   1  64  350gb 48:00 R 06:39 a035/0*64&lt;br /&gt;
3063.pbs        ley      epyc2    STDIN      766205   1  64  350gb 48:00 R 06:40 a036/0*64&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
In this table, you can see how many nodes and cores are being used by each job. For example, job 3029 of the user &amp;quot;ley&amp;quot; shows that it is running on 2 nodes using a total of 128 cores. In addition to the elapsed time, the table also show the reserved time for this job. This allow you to estimate when a job will be definitely finalized (or killed by the system if still running).&lt;br /&gt;
&lt;br /&gt;
The last column (without a header) is written because the option &#039;&#039;&#039;&amp;quot;-an1&amp;quot;&#039;&#039;&#039; was used. This useful to learn about which nodes are used by each job.&lt;br /&gt;
&lt;br /&gt;
===qstat -q===&lt;br /&gt;
&lt;br /&gt;
To learn more about the queues on the cluster, the &#039;&#039;&#039;&amp;quot;-q&amp;quot;&#039;&#039;&#039; option turns out to be useful.&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
$ qstat -q&lt;br /&gt;
&lt;br /&gt;
server: pbs&lt;br /&gt;
&lt;br /&gt;
Queue            Memory CPU Time Walltime Node   Run   Que   Lm  State&lt;br /&gt;
---------------- ------ -------- -------- ---- ----- ----- ----  -----&lt;br /&gt;
virtual            30gb    --       --       1     0     0   --   E R&lt;br /&gt;
a4000             500gb    --       --       1     0     0   --   E R&lt;br /&gt;
a6000             750gb    --       --       1     0     0   --   E R&lt;br /&gt;
xeon28             --      --       --       4     0     0   --   E R&lt;br /&gt;
amd16              --      --       --       8     0     1   --   E R&lt;br /&gt;
epyc2              --      --       --       2    14     0   --   E R&lt;br /&gt;
epyc1              --      --       --       2     0     0   --   E R&lt;br /&gt;
                                               ----- -----&lt;br /&gt;
                                                  14     1&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
For each queue, some basic values are displayed. The first three queues listed above have a default memory allocation as shown, and the column &#039;&#039;&#039;&amp;quot;Node&amp;quot;&#039;&#039;&#039; indicates the maximum number of nodes that can be asked for at job submission time. For example, you can request just one node for a job from the first three queues (because these are queues where MPI makes no sense). The &#039;&#039;&#039;&amp;quot;xeon28&amp;quot;&#039;&#039;&#039; queue also for a maximum of 4 nodes per MPI job. The &#039;&#039;&#039;&amp;quot;amd16&amp;quot;&#039;&#039;&#039; queue has a maximum of 8 nodes per job, and the &#039;&#039;&#039;&amp;quot;epyc1&amp;quot;&#039;&#039;&#039; and &#039;&#039;&#039;&amp;quot;epyc2&amp;quot;&#039;&#039;&#039; queues have maxima of two nodes per job. These limitations can be changed by the administrator as needed. As shown above, this will prevent inefficient resource requests.&lt;br /&gt;
&lt;br /&gt;
===qstat -f===&lt;br /&gt;
&lt;br /&gt;
The command &#039;&#039;&#039;&amp;quot;qstat -f -F json 3029&amp;quot;&#039;&#039;&#039; retrieves extremely detailed stats on the running job 3029. The result can be returned in JSON format to be ready for further processing (shown below).&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
$ qstat -f -F json 3029&lt;br /&gt;
{&lt;br /&gt;
    &amp;quot;timestamp&amp;quot;:1763705353,&lt;br /&gt;
    &amp;quot;pbs_version&amp;quot;:&amp;quot;23.06.06&amp;quot;,&lt;br /&gt;
    &amp;quot;pbs_server&amp;quot;:&amp;quot;pbs&amp;quot;,&lt;br /&gt;
    &amp;quot;Jobs&amp;quot;:{&lt;br /&gt;
        &amp;quot;3029.pbs&amp;quot;:{&lt;br /&gt;
            &amp;quot;Job_Name&amp;quot;:&amp;quot;STDIN&amp;quot;,&lt;br /&gt;
            &amp;quot;Job_Owner&amp;quot;:&amp;quot;ley@login4&amp;quot;,&lt;br /&gt;
            &amp;quot;resources_used&amp;quot;:{&lt;br /&gt;
                &amp;quot;cpupercent&amp;quot;:98,&lt;br /&gt;
                &amp;quot;cput&amp;quot;:&amp;quot;76:46:53&amp;quot;,&lt;br /&gt;
                &amp;quot;hpmem&amp;quot;:&amp;quot;0b&amp;quot;,&lt;br /&gt;
                &amp;quot;mem&amp;quot;:&amp;quot;52428800kb&amp;quot;,&lt;br /&gt;
                &amp;quot;ncpus&amp;quot;:128,&lt;br /&gt;
                &amp;quot;vmem&amp;quot;:&amp;quot;52428800kb&amp;quot;,&lt;br /&gt;
                &amp;quot;walltime&amp;quot;:&amp;quot;78:09:32&amp;quot;&lt;br /&gt;
            },&lt;br /&gt;
            &amp;quot;job_state&amp;quot;:&amp;quot;R&amp;quot;,&lt;br /&gt;
            &amp;quot;queue&amp;quot;:&amp;quot;epyc2&amp;quot;,&lt;br /&gt;
            &amp;quot;server&amp;quot;:&amp;quot;pbs&amp;quot;,&lt;br /&gt;
            &amp;quot;Checkpoint&amp;quot;:&amp;quot;u&amp;quot;,&lt;br /&gt;
            &amp;quot;ctime&amp;quot;:&amp;quot;Mon Nov 17 17:58:25 2025&amp;quot;,&lt;br /&gt;
            &amp;quot;Error_Path&amp;quot;:&amp;quot;/dev/pts/0&amp;quot;,&lt;br /&gt;
            &amp;quot;exec_host&amp;quot;:&amp;quot;a030/0*64+a031/0*64&amp;quot;,&lt;br /&gt;
            &amp;quot;exec_vnode&amp;quot;:&amp;quot;(a030:ncpus=64:mem=52428800kb)+(a031:ncpus=64:mem=52428800kb)&amp;quot;,&lt;br /&gt;
            &amp;quot;Hold_Types&amp;quot;:&amp;quot;n&amp;quot;,&lt;br /&gt;
            &amp;quot;interactive&amp;quot;:&amp;quot;True&amp;quot;,&lt;br /&gt;
            &amp;quot;Join_Path&amp;quot;:&amp;quot;n&amp;quot;,&lt;br /&gt;
            &amp;quot;Keep_Files&amp;quot;:&amp;quot;n&amp;quot;,&lt;br /&gt;
            &amp;quot;Mail_Points&amp;quot;:&amp;quot;a&amp;quot;,&lt;br /&gt;
            &amp;quot;mtime&amp;quot;:&amp;quot;Fri Nov 21 00:07:59 2025&amp;quot;,&lt;br /&gt;
            &amp;quot;Output_Path&amp;quot;:&amp;quot;/dev/pts/0&amp;quot;,&lt;br /&gt;
            &amp;quot;Priority&amp;quot;:0,&lt;br /&gt;
            &amp;quot;qtime&amp;quot;:&amp;quot;Mon Nov 17 17:58:25 2025&amp;quot;,&lt;br /&gt;
            &amp;quot;Rerunable&amp;quot;:&amp;quot;False&amp;quot;,&lt;br /&gt;
            &amp;quot;Resource_List&amp;quot;:{&lt;br /&gt;
                &amp;quot;mem&amp;quot;:&amp;quot;100gb&amp;quot;,&lt;br /&gt;
                &amp;quot;mpiprocs&amp;quot;:128,&lt;br /&gt;
                &amp;quot;ncpus&amp;quot;:128,&lt;br /&gt;
                &amp;quot;nodect&amp;quot;:2,&lt;br /&gt;
                &amp;quot;place&amp;quot;:&amp;quot;free&amp;quot;,&lt;br /&gt;
                &amp;quot;select&amp;quot;:&amp;quot;2:ncpus=64:mem=50gb:mpiprocs=64&amp;quot;,&lt;br /&gt;
                &amp;quot;walltime&amp;quot;:&amp;quot;200:00:00&amp;quot;&lt;br /&gt;
            },&lt;br /&gt;
            &amp;quot;stime&amp;quot;:&amp;quot;Mon Nov 17 17:58:25 2025&amp;quot;,&lt;br /&gt;
            &amp;quot;session_id&amp;quot;:2171964,&lt;br /&gt;
            &amp;quot;jobdir&amp;quot;:&amp;quot;/mnt/lustre/arrow/home/ley&amp;quot;,&lt;br /&gt;
            &amp;quot;substate&amp;quot;:42,&lt;br /&gt;
            &amp;quot;Variable_List&amp;quot;:{&lt;br /&gt;
                &amp;quot;PBS_O_HOME&amp;quot;:&amp;quot;/mnt/lustre/arrow/home/ley&amp;quot;,&lt;br /&gt;
                &amp;quot;PBS_O_LANG&amp;quot;:&amp;quot;en_US.UTF-8&amp;quot;,&lt;br /&gt;
                &amp;quot;PBS_O_LOGNAME&amp;quot;:&amp;quot;ley&amp;quot;,&lt;br /&gt;
                &amp;quot;PBS_O_PATH&amp;quot;:&amp;quot;/shared/apps/active/lstc/lsprepost/SP-4.5:/shared/apps/active/lstc/lsprepost/DP-4.3:/shared/bin:/usr/share/Modules/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/pbs/bin:/opt/thinlinc/bin:/opt/thinlinc/sbin:/mnt/lustre/arrow/home/ley/.local/bin:/mnt/lustre/arrow/home/ley/bin&amp;quot;,&lt;br /&gt;
                &amp;quot;PBS_O_MAIL&amp;quot;:&amp;quot;/var/spool/mail/ley&amp;quot;,&lt;br /&gt;
                &amp;quot;PBS_O_SHELL&amp;quot;:&amp;quot;/bin/bash&amp;quot;,&lt;br /&gt;
                &amp;quot;PBS_O_WORKDIR&amp;quot;:&amp;quot;/mnt/lustre/arrow/home/ley/Qualification/LS-Dyna/Rocky9/Seatbelt/Template&amp;quot;,&lt;br /&gt;
                &amp;quot;PBS_O_SYSTEM&amp;quot;:&amp;quot;Linux&amp;quot;,&lt;br /&gt;
                &amp;quot;PBS_O_QUEUE&amp;quot;:&amp;quot;epyc2&amp;quot;,&lt;br /&gt;
                &amp;quot;PBS_O_HOST&amp;quot;:&amp;quot;login4&amp;quot;&lt;br /&gt;
            },&lt;br /&gt;
            &amp;quot;comment&amp;quot;:&amp;quot;Job run at Mon Nov 17 at 17:58 on (a030:ncpus=64:mem=52428800kb)+(a031:ncpus=64:mem=52428800kb)&amp;quot;,&lt;br /&gt;
            &amp;quot;etime&amp;quot;:&amp;quot;Mon Nov 17 17:58:25 2025&amp;quot;,&lt;br /&gt;
            &amp;quot;run_count&amp;quot;:1,&lt;br /&gt;
            &amp;quot;Submit_arguments&amp;quot;:&amp;quot;-I -q epyc2 -l walltime=200:00:00,select=2:ncpus=64:mem=50gb:mpiprocs=64&amp;quot;,&lt;br /&gt;
            &amp;quot;project&amp;quot;:&amp;quot;_pbs_project_default&amp;quot;,&lt;br /&gt;
            &amp;quot;Submit_Host&amp;quot;:&amp;quot;login4&amp;quot;&lt;br /&gt;
        }&lt;br /&gt;
    }&lt;br /&gt;
}&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
===Manual pages for qstat===&lt;br /&gt;
&lt;br /&gt;
To learn more about the &#039;&#039;&#039;&amp;quot;qstat&amp;quot;&#039;&#039;&#039; command, you can use the command &#039;&#039;&#039;&amp;quot;man qstat&amp;quot;&#039;&#039;&#039;, which will print a lot of detailed information about the capabilities of this command.&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
$ man qstat&lt;br /&gt;
&lt;br /&gt;
qstat(1B)                                       PBS Professional                                       qstat(1B)&lt;br /&gt;
&lt;br /&gt;
NAME&lt;br /&gt;
       qstat - display status of PBS jobs, queues, or servers&lt;br /&gt;
&lt;br /&gt;
SYNOPSIS&lt;br /&gt;
       Displaying Job Status&lt;br /&gt;
              Default format:&lt;br /&gt;
              qstat [-E] [-J] [-p] [-t] [-w] [-x] [[&amp;lt;job ID&amp;gt; | &amp;lt;destination&amp;gt;] ...]&lt;br /&gt;
&lt;br /&gt;
              Long format:&lt;br /&gt;
              qstat -f [-F json | dsv [-D &amp;lt;delimiter&amp;gt;]] [-E] [-J] [-p] [-t] [-w]&lt;br /&gt;
                    [-x] [[&amp;lt;job ID&amp;gt; | &amp;lt;destination&amp;gt;] ...]&lt;br /&gt;
... &amp;lt;many more pages&amp;gt;&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
==Job Submission Basics==&lt;br /&gt;
&lt;br /&gt;
Jobs are submitted into the system using the &#039;&#039;&#039;&amp;quot;qsub&amp;quot;&#039;&#039;&#039; application. This application can take many different options and allows for a lot of different resource requests to tell the cluster what to do. We are running &#039;&#039;&#039;OpenPBS 23.06.06&#039;&#039;&#039; as our job scheduler. Here is a link to the User&#039;s Manual (of PBS PRO) if you want to explore gory details and capabilities. The User&#039;s Guide has about 240 pages, the Reference Guide has 500 pages, and the Big Book has 2500 pages. So there is a lot of information available. I also added job submission info for the LCRC cluster.&lt;br /&gt;
&lt;br /&gt;
* [https://argonne-lcrc.github.io/user-guides/running-jobs-at-lcrc/pbs-pro/ Argonne&#039;s LCRC pages on job submissions on their clusters]&lt;br /&gt;
* [https://help.altair.com/2022.1.0/PBS%20Professional/PBSUserGuide2022.1.pdf PBS Professional 2022.1 User&#039;s Guide]&lt;br /&gt;
* [https://help.altair.com/2022.1.0/PBS%20Professional/PBSReferenceGuide2022.1.pdf PBS Professional 2022.1 Reference Guide]&lt;br /&gt;
* [https://help.altair.com/2022.1.0/PBS%20Professional/PBS2022.1.pdf Altair PBS Professional 2022.1 Big Book]&lt;br /&gt;
&lt;br /&gt;
The User&#039;s Guide can be very helpful to clarify some of the concepts and capabilities, but it can be hard to find the specific information you may be looking for. Please understand that we are no longer running TORQUE and MAUI, so the syntax for job submission is distinctively different yet quite similar.&lt;br /&gt;
&lt;br /&gt;
The reference guide may be helpful to understand the complete syntax and full capabilities of the software.&lt;br /&gt;
&lt;br /&gt;
The big book is what I had to use when configuring OpenPBS earlier this year. This includes all the tricky details needed to make the system work smoothly for us. It&#039;s a bit scary to look at a PDF file that is 2500 pages long, but that is nothing compared to the StarCCM+ manuals.&lt;br /&gt;
&lt;br /&gt;
&amp;lt;BLOCKQUOTE&amp;gt;&lt;br /&gt;
[[File:Attention.jpg|25px]] &#039;&#039;&#039;IMPORTANT NOTE:&#039;&#039;&#039; &#039;&#039;The following sections are important to understand. They explain how jobs are submitted and then scjeduled for execution based on resources available and the specific need of the user.&#039;&#039;&lt;br /&gt;
&amp;lt;/BLOCKQUOTE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
The following sections explain the various tasks you may want to submit fir execution, ordered from simple to complex.&lt;br /&gt;
&lt;br /&gt;
* General Batch Jobs&lt;br /&gt;
** Requesting a Single Node for a Job&lt;br /&gt;
** Requesting Multiple Nodes for a Job&lt;br /&gt;
* Embedded Job Resource Requests&lt;br /&gt;
* Interactive Jobs&lt;br /&gt;
* Interactive Jobs with X-Windows GUI Applications&lt;br /&gt;
* Running Multiple Jobs on Single Nodes&lt;br /&gt;
* Running Jobs using GPUs&lt;br /&gt;
&lt;br /&gt;
===General Batch Jobs===&lt;br /&gt;
&lt;br /&gt;
Let&#039;s get started with a very basic usage of the system. Let&#039;s assume you have a simple application, and you want to execute it on a cluster node. Let&#039;s also assume that this is a very simple application, one that runs on one or a few cores, doesn&#039;t require any keyboard interaction with the user, doesn&#039;t need the user to see what&#039;s typically written to the screen, and writes its output to files. In this case, we can submit this application as a batch job, which will place it into an execution queue and process it as soon as a node becomes available.&lt;br /&gt;
&lt;br /&gt;
If the application requires more cores than a single node can provide, we can run the application over Infiniband with MPI message passing. In this case, we need to understand the concept of MPI applications a bit better. In both cases, we get started by creating a folder on the file system. Naming conventions are important so that you can distinguish the jobs by folder name.&lt;br /&gt;
&lt;br /&gt;
For both of the above scenarios, you would typically create a Bash shell script, and then submit the script into one of the queues for eventual execution.&lt;br /&gt;
&lt;br /&gt;
====Requesting a Single Node for a Job====&lt;br /&gt;
&lt;br /&gt;
Let&#039;s try something rather trivial to get used to the concept. Create yourself a folder, for example &#039;&#039;&#039;&amp;quot;myjobfolder&amp;quot;&#039;&#039;&#039;. Within that folder, create a job submission script. That script can have any name, but something short and simple may be best. Let&#039;s assume you create a file called &#039;&#039;&#039;&amp;quot;cluster.job&amp;quot;&#039;&#039;&#039;. The file doesn&#039;t have to have that extension. Any file name will do. But using the same filename for all of your jobs helps finding your way around the many files that will be created over time. The &#039;&#039;&#039;&amp;quot;cluster.job&amp;quot;&#039;&#039;&#039; file should look something like this:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
#!/bin/bash&lt;br /&gt;
#&lt;br /&gt;
# the following ensures that you will change into the directory where you are&lt;br /&gt;
# submitting the job from.&lt;br /&gt;
cd $PBS_O_WORKDIR&lt;br /&gt;
#&lt;br /&gt;
# now we sleep for 60 seconds and waste time. This is a placeholder for your application,&lt;br /&gt;
# which would be doing useful work for you.&lt;br /&gt;
sleep 60&lt;br /&gt;
#&lt;br /&gt;
# and after doing things, we may want to write something into a file to show that&lt;br /&gt;
# our jobs is done.&lt;br /&gt;
echo `date` &amp;gt; info.log&lt;br /&gt;
#&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
This can be submitted without detailed resource specifications (except for the walltime, which is be default 0:00:00):&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
$ qsub -q virtual -l walltime=1:00:00 cluster.job &lt;br /&gt;
3072.pbs&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
Wait a little, then check the status of running jobs:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
$ qstat -an1&lt;br /&gt;
&lt;br /&gt;
pbs: &lt;br /&gt;
                                                            Req&#039;d  Req&#039;d   Elap&lt;br /&gt;
Job ID          Username Queue    Jobname    SessID NDS TSK Memory Time  S Time&lt;br /&gt;
--------------- -------- -------- ---------- ------ --- --- ------ ----- - -----&lt;br /&gt;
3023.pbs        msitek   epyc2    STDIN      24360*   1  64  350gb 100:0 R 83:17 a028/0*64&lt;br /&gt;
3029.pbs        ley      epyc2    STDIN      21719*   2 128  100gb 200:0 R 74:00 a030/0*64+a031/0*64&lt;br /&gt;
3033.pbs        msitek   epyc2    STDIN      830486   1  64  350gb 100:0 R 59:23 a032/0*64&lt;br /&gt;
3048.pbs        james.c* amd16    foo.sh        --    1  28   30gb 01:00 Q   --   -- &lt;br /&gt;
3060.pbs        ley      epyc2    STDIN      763101   1  64  350gb 48:00 R 08:10 a033/0*64&lt;br /&gt;
3061.pbs        ley      epyc2    STDIN      763947   1  64  350gb 48:00 R 08:10 a034/0*64&lt;br /&gt;
3070.pbs        ley      epyc2    STDIN      766847   1  64  350gb 48:00 R 07:23 a042/0*64&lt;br /&gt;
3072.pbs        ley      virtual  cluster.j* 230230   1   4   30gb 01:00 E 00:01 v001/0*4&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
In this particular example, we are sending this job to the &#039;&#039;&#039;queue &amp;quot;virtual&amp;quot;&#039;&#039;&#039;. This queue, by default, allocates 30GB of memory to the job, and runs on 1 node with 4 cores. This is sufficient capacity to run quite a workload. When submitting a job to a single node, &#039;&#039;&#039;reasonable maximum allocations are automatically assigned&#039;&#039;&#039;, and the user doesn&#039;t have to worry about running out of memory or how many cores he will be using.&lt;br /&gt;
&lt;br /&gt;
The only required argument is the &#039;&#039;&#039;&amp;quot;walltime&amp;quot;&#039;&#039;&#039; argument. By default, the job will quit as soon as it is submitted. This indicates to the user that he forgot to provide the &#039;&#039;&#039;&amp;quot;walltime&amp;quot;&#039;&#039;&#039; argument.&lt;br /&gt;
&lt;br /&gt;
When the job disappears from the job list, it is done. At this point, you will find the file &amp;quot;info.log&amp;quot; in your job folder.&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
$ cat info.log &lt;br /&gt;
Thu Nov 20 08:00:31 PM CST 2025&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
====Requesting Multiple Nodes for a Job====&lt;br /&gt;
&lt;br /&gt;
To run jobs on multiple nodes, you will be likely &#039;&#039;&#039;executing jobs using MPI&#039;&#039;&#039;, the message passing interface. This establishes high-speed low-latency interconnections between the cores on one machine and the cores on the other machines. Data transfer does not require involvement of the cores themselves. Instead, the core tell the InfiniBand interconnect (and cores on the same node through shared memory) to transfer the data through RDMA, remoted direct memory access. The cores don&#039;t need to spend CPU cycles on copying data, but rather simply access the data once it has been copied by the Infiniband fabric. This makes for extremely efficient remote memory access, and message passing is used to coordinate data transfer between the cores no matter where they are located on any of the nodes.&lt;br /&gt;
&lt;br /&gt;
On our cluster, MPI-aware applications like &#039;&#039;&#039;OpenFOAM&#039;&#039;&#039;, &#039;&#039;&#039;StarCCM+&#039;&#039;&#039;, and &#039;&#039;&#039;LS-Dyna&#039;&#039;&#039; can be loaded as modules, which then automatically selects the most appropriate MPI library to use. The software applications have been tested to ensure that they work out-of-the box if a user selects any specific version of any of the applications.&lt;br /&gt;
&lt;br /&gt;
The following is a very trivial example for the MPI execution of a very simple executable, with one copy running on each core of the nodes allocated to  the job. It doesn&#039;t perform any real work and just wastes resources for a short time, but it illustrates how execution on the cores of various nodes works.&lt;br /&gt;
&lt;br /&gt;
Like in the previous section, we start with a simple job script that we submit to an appropriate queues. In this case, we pick a queue that has machines with Infiniband interfaces supporting efficient communications. Let&#039;s assume we edit a file with the name &#039;&#039;&#039;&amp;quot;parallel.job&amp;quot;&#039;&#039;&#039; like this:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
#!/bin/bash&lt;br /&gt;
#&lt;br /&gt;
# the following ensures that you will change into the directory where you are&lt;br /&gt;
# submitting the job from.&lt;br /&gt;
cd $PBS_O_WORKDIR&lt;br /&gt;
#&lt;br /&gt;
# to execute a simple command on all of the cores of all of the nodes allocated to the job,&lt;br /&gt;
# we need to make one of the MPI versions available. Let&#039;s use one of the most up-to-date&lt;br /&gt;
# MPI library available on the cluster&lt;br /&gt;
module load intel/2024.2.0/mpi/2021.13&lt;br /&gt;
#&lt;br /&gt;
# now we are apply a few settings that ensure that the MPI library will use the highest-performing&lt;br /&gt;
# Infiniband Interconnect, as well as a few options to tell MPI how to interface nodes with&lt;br /&gt;
# each other and which specific Infiniband adapter to use. This is complex and requires in-depth&lt;br /&gt;
# knowledge of the QLogic Infiniband adapters we are using. It is unlikely that you will ever have to&lt;br /&gt;
# deal with these options, because the &amp;quot;module load&amp;quot; command for the engineering applications we provide&lt;br /&gt;
# on ARROW will handle all those details transparently without the user needing to understand the details.&lt;br /&gt;
export I_MPI_HYDRA_BOOTSTRAP=ssh&lt;br /&gt;
export MPI_DEVICE=rdma:ofa-v2-ib0&lt;br /&gt;
export UCX_NET_DEVICES=qib0:1&lt;br /&gt;
#&lt;br /&gt;
# it doesn&#039;t make much sense, but in this example we are executing the OS command &amp;quot;uptime&amp;quot; on all cores&lt;br /&gt;
# of the nodes allocated to this job. The output from each core is written to the file info.log. We&lt;br /&gt;
# will find 56 lines of output in the file info.log, each created by the corresponding core executing&lt;br /&gt;
# the uptime command.&lt;br /&gt;
mpirun uptime &amp;gt; info.log&lt;br /&gt;
#&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
A good queue to test scripts is the &#039;&#039;&#039;&amp;quot;xeon28&amp;quot;&#039;&#039;&#039; queue. In the queue, we have 2 14-core Xeon processers per node, so that means that each node has 56 actual cores. We do not consider hyperthreading when doing parallel computing. 56 actual cores is what&#039;s being used here. The job submission will look like this:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
qsub -q xeon28 -l walltime=1:0:0 -l select=2:ncpus=28:mpiprocs=28:mem=60G parallel.job&lt;br /&gt;
        ^                  ^               ^       ^           ^      ^ ^ ^&lt;br /&gt;
        |                  |               |       |           |      | | + --- the name of the job script to execute&lt;br /&gt;
        |                  |               |       |           |      | + ----- don&#039;t forget to specify gigabytes&lt;br /&gt;
        |                  |               |       |           |      + ------- the amount of memory to request per node&lt;br /&gt;
        |                  |               |       |           + -------------- the number of MPI tasks per nodes&lt;br /&gt;
        |                  |               |       + -------------------------- the number of cores per node&lt;br /&gt;
        |                  |               + ---------------------------------- the number of nodes to select in the queue&lt;br /&gt;
        |                   + ------------------------------------------------- the requested time, in this case 1h&lt;br /&gt;
        + --------------------------------------------------------------------- the queue to be used for the job&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
At this point, the job has created a file &amp;quot;info.log&amp;quot; with 56 lines, one per core:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
 22:26:05 up 23 days,  9:28,  0 users,  load average: 0.00, 0.00, 0.00&lt;br /&gt;
 22:26:05 up 23 days,  9:28,  0 users,  load average: 0.00, 0.00, 0.00&lt;br /&gt;
 22:26:05 up 23 days,  9:28,  0 users,  load average: 0.00, 0.00, 0.00&lt;br /&gt;
 22:26:05 up 23 days,  9:28,  0 users,  load average: 0.00, 0.00, 0.00&lt;br /&gt;
 22:26:05 up 23 days,  9:28,  0 users,  load average: 0.00, 0.00, 0.00&lt;br /&gt;
 22:26:05 up 23 days,  9:28,  0 users,  load average: 0.00, 0.00, 0.00&lt;br /&gt;
 22:26:05 up 23 days,  9:53,  0 users,  load average: 0.06, 0.03, 0.00&lt;br /&gt;
 22:26:05 up 23 days,  9:53,  0 users,  load average: 0.06, 0.03, 0.00&lt;br /&gt;
 22:26:05 up 23 days,  9:53,  0 users,  load average: 0.06, 0.03, 0.00&lt;br /&gt;
...&lt;br /&gt;
&amp;lt;/PRE&amp;gt; &lt;br /&gt;
&lt;br /&gt;
In this simple example, the lines look all the same. Upon close examination through, you can find slightly different values for some of the lines. Some lines say that the machine is up for 23 days and 9:28, while others say 23 days and 9:53. Because all 28 cores of a node would see the same uptime of the server, half of the entries show one time stamp, and the other 28 cores show the other one. That demonstrates that the 56 processes have been running independently on 2 nodes.&lt;br /&gt;
&lt;br /&gt;
===Embedded Job Resource Requests===&lt;br /&gt;
&lt;br /&gt;
The job script can be modified to embed the resource requests in form of a series of &#039;&#039;&#039;#PBS&#039;&#039;&#039; statements at the beginning of the script file. This is a very common practice use at many HPC installations and job submission engines. Let&#039;s go back to the previous example where we run the script on two nodes in parallel. That is the &#039;&#039;&#039;&amp;quot;parallel.job&amp;quot;&#039;&#039;&#039; script file again:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
#!/bin/bash&lt;br /&gt;
#&lt;br /&gt;
#PBS -q xeon28&lt;br /&gt;
#PBS -l walltime=1:0:0&lt;br /&gt;
#PBS -l select=2:ncpus=28:mpiprocs=28:mem=60G&lt;br /&gt;
#&lt;br /&gt;
# the following ensures that you will change into the directory where you are&lt;br /&gt;
# submitting the job from.&lt;br /&gt;
cd $PBS_O_WORKDIR&lt;br /&gt;
#&lt;br /&gt;
# to execute a simple command on all of the cores of all of the nodes allocated to the job,&lt;br /&gt;
# we need to make one of the MPI versions available. Let&#039;s use one of the most up-to-date&lt;br /&gt;
# MPI library available on the cluster&lt;br /&gt;
module load intel/2024.2.0/mpi/2021.13&lt;br /&gt;
#&lt;br /&gt;
# now we are apply a few settings that ensure that the MPI library will use the highest-performing&lt;br /&gt;
# Infiniband Interconnect, as well as a few options to tell MPI how to interface nodes with&lt;br /&gt;
# each other and which specific Infiniband adapter to use. This is complex and requires in-depth&lt;br /&gt;
# knowledge of the QLogic Infiniband adapters we are using. It is unlikely that you will ever have to&lt;br /&gt;
# deal with these options, because the &amp;quot;module load&amp;quot; command for the engineering applications we provide&lt;br /&gt;
# on ARROW will handle all those details transparently without the user needing to understand the details.&lt;br /&gt;
export I_MPI_HYDRA_BOOTSTRAP=ssh&lt;br /&gt;
export MPI_DEVICE=rdma:ofa-v2-ib0&lt;br /&gt;
export UCX_NET_DEVICES=qib0:1&lt;br /&gt;
#&lt;br /&gt;
# it doesn&#039;t make much sense, but in this example we are executing the OS command &amp;quot;uptime&amp;quot; on all cores&lt;br /&gt;
# of the nodes allocated to this job. The output from each core is written to the file info.log. We&lt;br /&gt;
# will find 56 lines of output in the file info.log, each created by the corresponding core executing&lt;br /&gt;
# the uptime command.&lt;br /&gt;
mpirun uptime &amp;gt; info.log&lt;br /&gt;
#&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
If the resource requests are embedded within the file, they don&#039;t have to be specified on the command line any longer (the command line overrides the embedded specifications though). This may be convenient, because all the user has to do for job submission is the following:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
qsub parallel.job&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
Here is an example with more resource specifications and job settings that affect the behavior of the job:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
#!/bin/bash&lt;br /&gt;
#&lt;br /&gt;
#PBS -q xeon28&lt;br /&gt;
#PBS -l walltime=1:0:0&lt;br /&gt;
#PBS -l select=2:ncpus=28:mpiprocs=28:mem=60G&lt;br /&gt;
#PBS -A Account&lt;br /&gt;
#PBS -j oe&lt;br /&gt;
#PBS -N JobName&lt;br /&gt;
#PBS -e log.error&lt;br /&gt;
#PBS -o log.output&lt;br /&gt;
#PBS -m bae&lt;br /&gt;
#&lt;br /&gt;
...&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
I leave this to you as an exercise to figure out what the various options mean and how to specify them. There are many more, all documented in the PBS PRO manual (see above). Most of them are not terribly relevant and can be omitted.&lt;br /&gt;
&lt;br /&gt;
===Interactive Jobs===&lt;br /&gt;
&lt;br /&gt;
On ARROW, we don&#039;t restrict queues to be used only in batch mode. While &#039;&#039;&#039;batch mode&#039;&#039;&#039; is efficient for lining up a lot of work to be executed one after the other, ARROW has been designed to &#039;&#039;&#039;allow efficient model and software development in interactive mode&#039;&#039;&#039;. We have always ensured to have more computers than minimally needed to make it possible to dedicate resources to developers as needed, even if that means wasted CPU cycles. At times, we may ask you to limit the number of interactive jobs so that a large batch workload can be processed efficiently. This happens from time to time, and we have our users coordinate this with each other.&lt;br /&gt;
&lt;br /&gt;
Let&#039;s assume that you are developing an MPI application, or you are working on a complex &#039;&#039;&#039;OpenFOAM&#039;&#039;&#039; model that requires to start parallel processes over and over again just to find a bug and then fix it quickly. To do that, you can &#039;&#039;&#039;request an interactive job&#039;&#039;&#039; by adding the &#039;&#039;&#039;&amp;quot;-I&amp;quot;&#039;&#039;&#039; option to the job submission command (this is an uppercase I). Let&#039;s go to the parallel multi-node example from above:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
qsub -I -q xeon28 -l walltime=1:0:0 -l select=2:ncpus=28:mpiprocs=28:mem=60G&lt;br /&gt;
      ^    ^                  ^               ^       ^           ^      ^ ^&lt;br /&gt;
      |    |                  |               |       |           |      | + --- don&#039;t forget to specify gigabytes&lt;br /&gt;
      |    |                  |               |       |           |      + ----- the amount of memory to request per node&lt;br /&gt;
      |    |                  |               |       |           + ------------ the number of MPI tasks per nodes&lt;br /&gt;
      |    |                  |               |       + ------------------------ the number of cores per node&lt;br /&gt;
      |    |                  |               + -------------------------------- the number of nodes to select in the queue&lt;br /&gt;
      |    |                   + ----------------------------------------------- the requested time, in this case 1h&lt;br /&gt;
      |    + ------------------------------------------------------------------- the queue to be used for the job&lt;br /&gt;
      + ------------------------------------------------------------------------ request an interactive job &amp;lt;&amp;lt;===&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
When running interactive jobs with the &#039;&#039;&#039;&amp;quot;-I&amp;quot;&#039;&#039;&#039; parameter, we don&#039;t specify av job script at the end of the submission command. The interactive job will instead start (once the nodes are available) in interactive mode, meaning that the terminal session changes over from being a series of commands executed on the login server to being a series of commands being executed on the first node of the group of nodes that are allocated to the job. At this point, you can change to the desired working directory, but what you do with the allocated resources is entirely up to you. You can load modules, including MPI libraries, and then issue the commands for your application interactively and see how they execute. If you start an &#039;&#039;&#039;&amp;quot;mpirun&amp;quot;&#039;&#039;&#039;, the cores on your allocated secondary node will work as expected. There is no difference to batch mode, other than you having the ability to execute lines of commands at will.&lt;br /&gt;
&lt;br /&gt;
===Interactive Jobs with X-Windows GUI Applications===&lt;br /&gt;
&lt;br /&gt;
Interactive use can go further than that. With some of our software applications, like &#039;&#039;&#039;StarCCM+&#039;&#039;&#039;, you can run an &#039;&#039;&#039;interactive GUI application&#039;&#039;&#039; where you control the computational work from within the applications&#039; GUI. Within the GUI, you can control execution of the numerical solver that runs on as many cores as you requested, while being able to reconfigure the case through the GUI as well. Furthermore, you can visualize developing results on the fly by creating complex plots and visualizations.&lt;br /&gt;
&lt;br /&gt;
All that is need is an option &#039;&#039;&#039;&amp;quot;-X&amp;quot;&#039;&#039;&#039; being used as part of the job submission, like this:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
qsub -X -I -q xeon28 -l walltime=1:0:0 -l select=2:ncpus=28:mpiprocs=28:mem=60G&lt;br /&gt;
      ^  ^    ^                  ^               ^       ^           ^      ^ ^&lt;br /&gt;
      |  |    |                  |               |       |           |      | + --- don&#039;t forget to specify gigabytes&lt;br /&gt;
      |  |    |                  |               |       |           |      + ----- the amount of memory to request per node&lt;br /&gt;
      |  |    |                  |               |       |           + ------------ the number of MPI tasks per nodes&lt;br /&gt;
      |  |    |                  |               |       + ------------------------ the number of cores per node&lt;br /&gt;
      |  |    |                  |               + -------------------------------- the number of nodes to select in the queue&lt;br /&gt;
      |  |    |                   + ----------------------------------------------- the requested time, in this case 1h&lt;br /&gt;
      |  |    + ------------------------------------------------------------------- the queue to be used for the job&lt;br /&gt;
      |  + ------------------------------------------------------------------------ request an interactive job&lt;br /&gt;
      + --------------------------------------------------------------------------- request GUI capabilities &amp;lt;&amp;lt;===&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
===Running Multiple Jobs on Single Nodes===&lt;br /&gt;
&lt;br /&gt;
A feature that is new on ARROW is the ability to run multiple jobs on a single node. Let&#039;s assume that you are performing a sensitivity analysis on an existing model, and the model is simple enough to return results within a reasonable time on just a few cores of a higher end machine (maybe you are running SMP versions of &#039;&#039;&#039;LS-Dyna&#039;&#039;&#039;). Our high end machines have 64 cores, so lets assume you have an &#039;&#039;&#039;LS-Dyna&#039;&#039;&#039; model that runs well on 8 cores and doesn&#039;t use a whole lot of memory. In this case, you can submit individual jobs that request simply 8 cores and a fraction of the available memory available on the node, and all jobs execute independently from each other. Each job is fit into a slot where available. It is not very different from using whole nodes for everything. The important consideration is that each job is cleanly constrained into it&#039;s allotted resources using the &#039;&#039;&#039;CGROUPS&#039;&#039;&#039; functionality of modern operating systems. Because an abusive user cannot use more cores or more memory than allocated to his job, other users can safely run smaller jobs on the same node.&lt;br /&gt;
&lt;br /&gt;
Lets assume that we have a number of smaller jobs that we want to run on a single node in the &#039;&#039;&#039;&amp;quot;xeon28&amp;quot;&#039;&#039;&#039; queue. Each job would be submitted by using reduced resources that allow for sharing but that  guarantee that the jobs will be run successfully. In this case, you can &#039;&#039;&#039;submit many jobs&#039;&#039;&#039; in the following manner (with a job script for the small jobs, each of which can &#039;&#039;&#039;request varying resources&#039;&#039;&#039; if needed - some may want to run on 5 cores, others on 3):&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
#!/bin/bash&lt;br /&gt;
#&lt;br /&gt;
# the following ensures that you will change into the directory where you are&lt;br /&gt;
# submitting the job from.&lt;br /&gt;
cd $PBS_O_WORKDIR&lt;br /&gt;
#&lt;br /&gt;
# now we sleep for 300 seconds and waste time. This is a placeholder for your application,&lt;br /&gt;
# which would be doing useful work for you.&lt;br /&gt;
sleep 300&lt;br /&gt;
#&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
Now we submit a variety of these jobs (11 total in this example) to the &#039;&#039;&#039;&amp;quot;xeon28&amp;quot;&#039;&#039;&#039; queue for execution (note that the first few jobs request different amounts of memory and core counts):&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
qsub -q xeon28 -l walltime=1:0:0 -l select=1:ncpus=12:mpiprocs=12:mem=5G small.job &lt;br /&gt;
qsub -q xeon28 -l walltime=1:0:0 -l select=1:ncpus=10:mpiprocs=10:mem=7G small.job &lt;br /&gt;
qsub -q xeon28 -l walltime=1:0:0 -l select=1:ncpus=8:mpiprocs=8:mem=9G small.job &lt;br /&gt;
qsub -q xeon28 -l walltime=1:0:0 -l select=1:ncpus=16:mpiprocs=16:mem=20G small.job &lt;br /&gt;
qsub -q xeon28 -l walltime=1:0:0 -l select=1:ncpus=2:mpiprocs=2:mem=2G small.job &lt;br /&gt;
qsub -q xeon28 -l walltime=1:0:0 -l select=1:ncpus=2:mpiprocs=2:mem=2G small.job &lt;br /&gt;
qsub -q xeon28 -l walltime=1:0:0 -l select=1:ncpus=2:mpiprocs=2:mem=2G small.job &lt;br /&gt;
qsub -q xeon28 -l walltime=1:0:0 -l select=1:ncpus=2:mpiprocs=2:mem=2G small.job &lt;br /&gt;
qsub -q xeon28 -l walltime=1:0:0 -l select=1:ncpus=2:mpiprocs=2:mem=2G small.job &lt;br /&gt;
qsub -q xeon28 -l walltime=1:0:0 -l select=1:ncpus=2:mpiprocs=2:mem=2G small.job &lt;br /&gt;
qsub -q xeon28 -l walltime=1:0:0 -l select=1:ncpus=2:mpiprocs=2:mem=2G small.job &lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
They are now running in the order of submission, allocated on as few nodes in the &amp;quot;xeon28&amp;quot; queue as necessary. Only 2 nodes are being loaded quite heavily, and 4 more cores are in use on a third node.&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
Nov 20 23:34 ley@login3:myjobfolder$ qstat -an1&lt;br /&gt;
&lt;br /&gt;
pbs: &lt;br /&gt;
                                                            Req&#039;d  Req&#039;d   Elap&lt;br /&gt;
Job ID          Username Queue    Jobname    SessID NDS TSK Memory Time  S Time&lt;br /&gt;
--------------- -------- -------- ---------- ------ --- --- ------ ----- - -----&lt;br /&gt;
&lt;br /&gt;
3082.pbs        ley      xeon28   small.job  813221   1  12    5gb 01:00 R 00:01 p001/0*12&lt;br /&gt;
3083.pbs        ley      xeon28   small.job  813288   1  10    7gb 01:00 R 00:01 p001/1*10&lt;br /&gt;
3084.pbs        ley      xeon28   small.job  671792   1   8    9gb 01:00 R 00:01 p002/0*8&lt;br /&gt;
3085.pbs        ley      xeon28   small.job  671845   1  16   20gb 01:00 R 00:01 p002/1*16&lt;br /&gt;
3086.pbs        ley      xeon28   small.job  813361   1   2    2gb 01:00 R 00:00 p001/2*2&lt;br /&gt;
3087.pbs        ley      xeon28   small.job  813413   1   2    2gb 01:00 R 00:00 p001/3*2&lt;br /&gt;
3088.pbs        ley      xeon28   small.job  813464   1   2    2gb 01:00 R 00:00 p001/4*2&lt;br /&gt;
3089.pbs        ley      xeon28   small.job  671912   1   2    2gb 01:00 R 00:00 p002/2*2&lt;br /&gt;
3090.pbs        ley      xeon28   small.job  671969   1   2    2gb 01:00 R 00:00 p002/3*2&lt;br /&gt;
3091.pbs        ley      xeon28   small.job  632092   1   2    2gb 01:00 R 00:00 p003/0*2&lt;br /&gt;
3092.pbs        ley      xeon28   small.job  632100   1   2    2gb 01:00 R 00:00 p003/1*2&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
This is a particularly effective strategy to run concurrently many cases that don&#039;t scale well beyond a few cores. When running them on fewer cores but many of them at the same time, the overall processing rate will be much higher than executing them the traditional way.&lt;br /&gt;
&lt;br /&gt;
===Running Jobs using GPUs===&lt;br /&gt;
&lt;br /&gt;
The principle of running multiple jobs on a single node becomes particularly important when using servers equipped with &#039;&#039;&#039;GPUs&#039;&#039;&#039; for &#039;&#039;&#039;ML/AI&#039;&#039;&#039; applications. The cluster doesn&#039;t have a whole lot of &#039;&#039;&#039;GPUs&#039;&#039;&#039; at this point. We have three machines with three &#039;&#039;&#039;A4000&#039;&#039;&#039; GOUs, a &#039;&#039;&#039;total of 9 A4000 GPUs&#039;&#039;&#039;. Then we have a much more powerful single machine with our &#039;&#039;&#039;four A6000 GPUs&#039;&#039;&#039;.&lt;br /&gt;
&lt;br /&gt;
Using multiple GPUs in a single application is still something where the software has to be designed with hardware in mind. GPUs have several methods of communicating with each other, e.g. very fast &#039;&#039;&#039;NVLINK&#039;&#039;&#039; between pairs of GPUs, GPUs being directly connected to one of the two CPUs in the system and thus being able to communicate faster, and GPUs that have to jump between processors when communicating, and then the whole issue of having to go possibly through PCIe bridges.&lt;br /&gt;
&lt;br /&gt;
On our system, we are providing the ability to &#039;&#039;&#039;work mostly with individual GPUs&#039;&#039;&#039;. Users can also reserve entire nodes and develop or run applications that are adapted to that hardware, including several GPUs installed on that node. One thing we do not provide is the ability of GPU to GPU communication between nodes. Thus, a job cannot request more than one GPU node at a time.&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
qsub -q a4000 -I -l walltime=1:0:0 -l select=1:ncpus=5:mem=150G:ngpus=1&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
With these specifications, three single GPU jobs can run on a single server. Each job sees only one of the reserved GPU.&lt;br /&gt;
&lt;br /&gt;
To run a massive GPU job on 64 cores with 4 &#039;&#039;&#039;A6000 GPUs&#039;&#039;&#039;, submit the job like this:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
qsub -q a6000 -I -l walltime=1:0:0 -l select=1:ncpus=64:mem=725G:ngpus=4&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
===Manual pages for qsub===&lt;br /&gt;
&lt;br /&gt;
To learn more about the &amp;quot;qsub&amp;quot; command, you can use the command &amp;quot;man qsub&amp;quot;, which will print a lot of detailed information about the capabilities of this command.&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
$ man qsub&lt;br /&gt;
&lt;br /&gt;
qsub(1B)                                               PBS Professional                                              qsub(1B)&lt;br /&gt;
&lt;br /&gt;
NAME&lt;br /&gt;
       qsub - submit a job to PBS&lt;br /&gt;
&lt;br /&gt;
SYNOPSIS&lt;br /&gt;
       qsub [-a &amp;lt;date and time&amp;gt;] [-A &amp;lt;account string&amp;gt;] [-c &amp;lt;checkpoint spec&amp;gt;]&lt;br /&gt;
            [-C &amp;lt;directive prefix&amp;gt;] [-e &amp;lt;path&amp;gt;] [-f] [-h]&lt;br /&gt;
            [-I [-G [-- &amp;lt;GUI application/script&amp;gt;]] | [-X]] [-j &amp;lt;join&amp;gt;]&lt;br /&gt;
            [-J &amp;lt;range&amp;gt; [%&amp;lt;max subjobs]] [-k &amp;lt;discard&amp;gt;] [-l &amp;lt;resource list&amp;gt;]&lt;br /&gt;
            [-m &amp;lt;mail events&amp;gt;] [-M &amp;lt;user list&amp;gt;] [-N &amp;lt;name&amp;gt;] [-o &amp;lt;path&amp;gt;]&lt;br /&gt;
            [-p &amp;lt;priority&amp;gt;] [-P &amp;lt;project&amp;gt;] [-q &amp;lt;destination&amp;gt;] [-r &amp;lt;y|n&amp;gt;]&lt;br /&gt;
            [-R &amp;lt;remove options&amp;gt;] [-S &amp;lt;path list&amp;gt;] [-u &amp;lt;user list&amp;gt;]&lt;br /&gt;
            [-v &amp;lt;variable list&amp;gt;] [-V] [-W &amp;lt;additional attributes&amp;gt;] [-z]&lt;br /&gt;
            [- | &amp;lt;script&amp;gt; | -- &amp;lt;executable&amp;gt; [&amp;lt;arguments to executable&amp;gt;]]&lt;br /&gt;
       qsub --version&lt;br /&gt;
&lt;br /&gt;
DESCRIPTION&lt;br /&gt;
       You use the qsub command to submit a batch job to PBS.  Submitting a PBS job specifies a task, requests resources, and&lt;br /&gt;
       sets job attributes.&lt;br /&gt;
... &amp;lt;many more pages&amp;gt;&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
==LS-Dyna on the ARROW Cluster==&lt;br /&gt;
&lt;br /&gt;
===Currently Available LS-Dyna Versions===&lt;br /&gt;
&lt;br /&gt;
The following is a list of &#039;&#039;&#039;LS-Dyna versions&#039;&#039;&#039; available on &#039;&#039;&#039;ARROW&#039;&#039;&#039; after the latest reconfiguration of the system. As per LSTC/ANSYS, &#039;&#039;&#039;versions before 14.0.0 are not necessarily fully supported any longer&#039;&#039;&#039; because they are supposedly not compatible with modern operating systems and cannot be made to work reliably. We have tested the listed older versions of LS-Dyna and they have passed basic tests. They may not behave exactly as they did on the old CentOS 7 operating system, and time will show whether they can still be used or whether you will need to update your models and use a fully supported version.&lt;br /&gt;
&lt;br /&gt;
All versions are loaded using the &#039;&#039;&#039;&amp;quot;module load&amp;quot;&#039;&#039;&#039; command. Versions can be listed with the &#039;&#039;&#039;&amp;quot;module avail ls-dyna&amp;quot;&#039;&#039;&#039; command. To load one of the modules, use the following syntax:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
module load ls-dyna/14.2.0/mpi-d8-ifort190-avx512&lt;br /&gt;
            ^       ^      ^   ^  ^        ^&lt;br /&gt;
            |       |      |   |  |        + --- specify the extended instruction set needed for execution&lt;br /&gt;
            |       |      |   |  + ------------ load the version of the compiler that was used to create this &lt;br /&gt;
            |       |      |   + --------------- load the version that supports double precision variables&lt;br /&gt;
            |       |      + ------------------- load the MPP (MPI) version of LS-Dyna&lt;br /&gt;
            |       + -------------------------- load specifically version 14.2.0&lt;br /&gt;
            + ---------------------------------- load a version of LS-Dyna&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
The version string is composed of multiple elements to indicate variants in compilers and compiler options. Use the following guideline to choose an appropriate version to load:&lt;br /&gt;
&lt;br /&gt;
* &#039;&#039;&#039;&amp;quot;1&amp;quot;&#039;&#039;&#039; or &#039;&#039;&#039;&amp;quot;mpi&amp;quot;&#039;&#039;&#039; indicates whether this is a single node version of LS-Dyna (&#039;&#039;&#039;SMP&#039;&#039;&#039;) or whether this is a multi-node MPI version (&#039;&#039;&#039;MPP&#039;&#039;&#039;). All MPI versions use the &#039;&#039;&#039;IntelMPI 2022&#039;&#039;&#039; libraries which have been tested thoroughly on ARROW. MPI versions will use the Infiniband Network of ARROW for high-speed and low-latency inter-process communication using RDMA (remote direct memory access).&lt;br /&gt;
* All LS-Dyna versions are available in either &#039;&#039;&#039;floating point&#039;&#039;&#039; or &#039;&#039;&#039;double precision variants&#039;&#039;&#039;. Floating point variants use 4 bytes to represent a value, and double precision variants use 8 bytes. There are pros and cons for choosing one over the other variant. With regards to computational efficiency, both perform nearly the same because all machines are equipped with 64-bit CPUs.&lt;br /&gt;
** &#039;&#039;&#039;&amp;quot;f4&amp;quot;&#039;&#039;&#039; floating point versions&lt;br /&gt;
*** Pros: These require significantly less memory to run. Results occupy less disk space, and can be transferred significantly faster into and out of ARROW.&lt;br /&gt;
*** Cons: The numerical resolution is limited to 7 significant digits, which is often undesirable when dealing with mathematical operations on small and large numbers at the same time.&lt;br /&gt;
** &#039;&#039;&#039;&amp;quot;r8&amp;quot;&#039;&#039;&#039; double precision versions&lt;br /&gt;
*** Pros: The numerical resolution is about twice the number of significant digits compare to &amp;quot;f4&amp;quot;, which helps when when dealing with mathematical operations on small and large numbers at the same time.&lt;br /&gt;
*** Cons: These require more memory to run. Results occupy more disk space, and it takes longer to transfer data into and out of ARROW.&lt;br /&gt;
* There are two more identifiers to choose from when it comes to the variants of the executables: the specific compiler used to create the executable and the specific processor instruction set required for running the executable.&lt;br /&gt;
** For modern versions of LS-Dyna, two compilers have been used by the developers to create LS-Dyna executables: the &#039;&#039;&#039;Intel Fortran Compiler&#039;&#039;&#039; and the &#039;&#039;&#039;AOCC (AMD Optimizing C/C++ and Fortran) compiler&#039;&#039;&#039;. Both variants of the software are supported on ARROW. This gives users the opportunity to choose an alternate variant of the same LS-Dyna version when running into bugs or crashes.&lt;br /&gt;
** The variants based on the various instruction set extensions (&#039;&#039;&#039;SSE2&#039;&#039;&#039;, &#039;&#039;&#039;AVX2&#039;&#039;&#039;, &#039;&#039;&#039;AVX512&#039;&#039;&#039;, and so on) gives users even more options when choosing an alternate LS-Dyna variant of the same version when running into bugs or crashes. These instruction sets are mostly related to performance gains on specific processors. We have not performed thorough performance tests and cannot recommend specific versions right now.&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
$ module avail ls-dyna&lt;br /&gt;
--------------------------------------------- /shared/apps/modulefiles ---------------------------------------------&lt;br /&gt;
ls-dyna/09.3.1/1-d8-ifort131           ls-dyna/12.2.1/mpi-f4-ifort160-sse2    ls-dyna/14.2.0/mpi-f4-ifort190-avx512  &lt;br /&gt;
ls-dyna/09.3.1/1-f4-ifort131           ls-dyna/12.2.2/1-d8-ifort160-sse2      ls-dyna/14.2.0/mpi-f4-ifort190-sse2    &lt;br /&gt;
ls-dyna/09.3.1/mpi-d8-ifort131-avx2    ls-dyna/12.2.2/1-f4-ifort160-sse2      ls-dyna/15.0.2/1-d8-ifort190-sse2      &lt;br /&gt;
ls-dyna/09.3.1/mpi-d8-ifort131-avx512  ls-dyna/12.2.2/mpi-d8-aocc400-avx2     ls-dyna/15.0.2/1-f4-ifort190-sse2      &lt;br /&gt;
ls-dyna/09.3.1/mpi-f4-ifort131-avx2    ls-dyna/12.2.2/mpi-d8-ifort160-avx2    ls-dyna/15.0.2/mpi-d8-aocc400-avx2     &lt;br /&gt;
ls-dyna/09.3.1/mpi-f4-ifort131-avx512  ls-dyna/12.2.2/mpi-d8-ifort160-sse2    ls-dyna/15.0.2/mpi-d8-ifort190-avx2    &lt;br /&gt;
ls-dyna/10.2.0/1-d8-ifort160           ls-dyna/12.2.2/mpi-f4-aocc400-avx2     ls-dyna/15.0.2/mpi-d8-ifort190-avx512  &lt;br /&gt;
ls-dyna/10.2.0/1-f4-ifort160           ls-dyna/12.2.2/mpi-f4-ifort160-avx2    ls-dyna/15.0.2/mpi-d8-ifort190-sse2    &lt;br /&gt;
ls-dyna/11.0.0/1-d8-ifort160           ls-dyna/12.2.2/mpi-f4-ifort160-sse2    ls-dyna/15.0.2/mpi-f4-aocc400-avx2     &lt;br /&gt;
ls-dyna/11.0.0/1-f4-ifort160           ls-dyna/13.0.0/1-d8-ifort190           ls-dyna/15.0.2/mpi-f4-ifort190-avx2    &lt;br /&gt;
ls-dyna/11.1.0/1-d8-ifort160-sse2      ls-dyna/13.0.0/1-f4-ifort190           ls-dyna/15.0.2/mpi-f4-ifort190-avx512  &lt;br /&gt;
ls-dyna/11.1.0/1-f4-ifort160-sse2      ls-dyna/13.0.0/mpi-d8-ifort190-avx2    ls-dyna/15.0.2/mpi-f4-ifort190-sse2    &lt;br /&gt;
ls-dyna/11.2.0/1-d8-ifort160           ls-dyna/13.0.0/mpi-d8-ifort190-sse2    ls-dyna/16.0.0/1-d8-aocc420-avx2       &lt;br /&gt;
ls-dyna/11.2.0/1-f4-ifort160           ls-dyna/13.0.0/mpi-f4-ifort190-avx2    ls-dyna/16.0.0/1-d8-aocc420-avx512     &lt;br /&gt;
ls-dyna/11.2.0/mpi-f4-ifort160-avx2    ls-dyna/13.0.0/mpi-f4-ifort190-sse2    ls-dyna/16.0.0/1-d8-ifort190-sse2      &lt;br /&gt;
ls-dyna/11.2.0/mpi-f4-ifort160-sse2    ls-dyna/13.1.0/mpi-d8-aocc310-avx2     ls-dyna/16.0.0/1-f4-aocc420-avx2       &lt;br /&gt;
ls-dyna/11.2.1/1-d8-ifort160           ls-dyna/13.1.0/mpi-d8-ifort190-avx2    ls-dyna/16.0.0/1-f4-aocc420-avx512     &lt;br /&gt;
ls-dyna/11.2.1/1-f4-ifort160           ls-dyna/13.1.0/mpi-d8-ifort190-sse2    ls-dyna/16.0.0/1-f4-ifort190-sse2      &lt;br /&gt;
ls-dyna/11.2.1/mpi-d8-ifort160-avx2    ls-dyna/13.1.0/mpi-f4-aocc310-avx2     ls-dyna/16.0.0/mpi-d8-aocc420-avx2     &lt;br /&gt;
ls-dyna/11.2.1/mpi-d8-ifort160-sse2    ls-dyna/13.1.0/mpi-f4-ifort190-avx2    ls-dyna/16.0.0/mpi-d8-aocc420-avx512   &lt;br /&gt;
ls-dyna/11.2.1/mpi-f4-ifort160-avx2    ls-dyna/13.1.0/mpi-f4-ifort190-sse2    ls-dyna/16.0.0/mpi-d8-ifort190-avx2    &lt;br /&gt;
ls-dyna/11.2.1/mpi-f4-ifort160-sse2    ls-dyna/13.1.1/mpi-d8-ifort190-avx2    ls-dyna/16.0.0/mpi-d8-ifort190-avx512  &lt;br /&gt;
ls-dyna/11.2.2/mpi-d8-ifort160-avx2    ls-dyna/13.1.1/mpi-d8-ifort190-sse2    ls-dyna/16.0.0/mpi-d8-ifort190-sse2    &lt;br /&gt;
ls-dyna/11.2.2/mpi-d8-ifort160-sse2    ls-dyna/13.1.1/mpi-f4-ifort190-avx2    ls-dyna/16.0.0/mpi-f4-aocc420-avx2     &lt;br /&gt;
ls-dyna/11.2.2/mpi-f4-ifort160-avx2    ls-dyna/13.1.1/mpi-f4-ifort190-sse2    ls-dyna/16.0.0/mpi-f4-aocc420-avx512   &lt;br /&gt;
ls-dyna/11.2.2/mpi-f4-ifort160-sse2    ls-dyna/14.0.0/1-d8-ifort190           ls-dyna/16.0.0/mpi-f4-ifort190-avx2    &lt;br /&gt;
ls-dyna/12.1.0/1-d8-ifort160           ls-dyna/14.0.0/1-f4-ifort190           ls-dyna/16.0.0/mpi-f4-ifort190-avx512  &lt;br /&gt;
ls-dyna/12.1.0/1-f4-aocc310            ls-dyna/14.0.0/mpi-d8-aocc310-avx2     ls-dyna/16.0.0/mpi-f4-ifort190-sse2    &lt;br /&gt;
ls-dyna/12.1.0/1-f4-ifort160           ls-dyna/14.0.0/mpi-d8-ifort190-avx2    ls-dyna/16.1.0/mpi-d8-aocc420-avx2     &lt;br /&gt;
ls-dyna/12.1.0/mpi-d8-aocc310-avx2     ls-dyna/14.0.0/mpi-d8-ifort190-sse2    ls-dyna/16.1.0/mpi-d8-aocc420-avx512   &lt;br /&gt;
ls-dyna/12.1.0/mpi-d8-ifort160-avx2    ls-dyna/14.0.0/mpi-f4-ifort190-avx2    ls-dyna/16.1.0/mpi-d8-ifort190-avx2    &lt;br /&gt;
ls-dyna/12.1.0/mpi-d8-ifort160-sse2    ls-dyna/14.0.0/mpi-f4-ifort190-sse2    ls-dyna/16.1.0/mpi-d8-ifort190-avx512  &lt;br /&gt;
ls-dyna/12.1.0/mpi-f4-aocc310-avx2     ls-dyna/14.1.0/1-d8-ifort190-sse2      ls-dyna/16.1.0/mpi-d8-ifort190-sse2    &lt;br /&gt;
ls-dyna/12.1.0/mpi-f4-ifort160-avx2    ls-dyna/14.1.0/1-f4-ifort190-sse2      ls-dyna/16.1.0/mpi-f4-aocc420-avx2     &lt;br /&gt;
ls-dyna/12.1.0/mpi-f4-ifort160-sse2    ls-dyna/14.1.0/mpi-d8-aocc400-avx2     ls-dyna/16.1.0/mpi-f4-aocc420-avx512&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
===Submitting an LS-Dyna Job===&lt;br /&gt;
&lt;br /&gt;
&amp;lt;BLOCKQUOTE&amp;gt;&lt;br /&gt;
[[File:Attention.jpg|25px]] &#039;&#039;&#039;IMPORTANT NOTE:&#039;&#039;&#039; The job/queue manager can now track the number of LS-Dyna licenses given out to individual&lt;br /&gt;
jobs. At submission time, it is not possible to know what software a user may run. But by adding the clause &amp;quot;-l dynalic&amp;quot; at submission time,&lt;br /&gt;
the queue manager can calculate the total number of cores required and keep track of LS-Dyna licenses used by the job. When loading a version of LS-Dyna, a check will be performed, and LS-Dyna will be prevented from running if the &amp;quot;-l dynalic&amp;quot; clause was not used when submitting the job.&#039;&#039;&lt;br /&gt;
&lt;br /&gt;
&amp;lt;!--&lt;br /&gt;
&#039;&#039;The job/queue manager can track the number of LS-Dyna licenses to some degree. If all &#039;&#039;&#039;LS-Dyna users&#039;&#039;&#039; cooperate and use a script like the one shown below when submitting their jobs, the total number of concurrent &#039;&#039;&#039;LS-Dyna licenses&#039;&#039;&#039; will be tracked by the job manager correctly. That means that users can submit any number of LS-Dyna jobs, and jobs will only start when a sufficient number of licenses is available. This is managed by the &#039;&#039;&#039;&amp;quot;dynalic&amp;quot;&#039;&#039;&#039; resource at the end of the select statement. In this example, a 2-node job on 64-core nodes will need a total of &#039;&#039;&#039;&amp;quot;dynalic=128&amp;quot;&#039;&#039;&#039; licenses. This accounting breaks down when users don&#039;t use the &#039;&#039;&#039;&amp;quot;dynalic=XXX&amp;quot;&#039;&#039;&#039; statement, or when they don&#039;t calculate the number of licenses correctly. In that case, LS-Dyna jobs of all users are subject to sudden failure when LS-Dyna licenses run out. Please understand the importance of this specific setting in your job.&#039;&#039;&lt;br /&gt;
--&amp;gt;&lt;br /&gt;
&amp;lt;/BLOCKQUOTE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
Furthermore, careful consideration should be given with regards to choice of resources for an &#039;&#039;&#039;LS-Dyna job&#039;&#039;&#039;. With 64 cores available on a single node in the &#039;&#039;&#039;&amp;quot;epyc1&amp;quot;&#039;&#039;&#039; and &#039;&#039;&#039;&amp;quot;epyc2&amp;quot;&#039;&#039;&#039; queues, it may be counterproductive to run a job on two nodes instead of a single node. Users should run their jobs with different numbers of nodes and determine whether performance increases. It may well decrease when running a job on two or more nodes. The outcome of such tests will tell what the best allocation of resources will be.&lt;br /&gt;
&lt;br /&gt;
Most users use a job script like the following. All methods for job submission the the previous chapters apply as well, so there is a lot of flexibility:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
#!/bin/bash&lt;br /&gt;
#&lt;br /&gt;
#PBS -q epyc1&lt;br /&gt;
#PBS -l walltime=12:0:0&lt;br /&gt;
#PBS -l select=2:ncpus=64:mpiprocs=64:mem=225G,dynalic&lt;br /&gt;
#PBS -N JobName&lt;br /&gt;
#PBS -e log.error&lt;br /&gt;
#PBS -o log.output&lt;br /&gt;
#&lt;br /&gt;
cd $PBS_O_WORKDIR&lt;br /&gt;
#&lt;br /&gt;
module load ls-dyna/12.2.1/mpi-f4-ifort160-avx2&lt;br /&gt;
module load dynamore/current&lt;br /&gt;
#&lt;br /&gt;
mpirun ls-dyna i=main.k memory1=300m memory2=100m &amp;gt; dyna.log&lt;br /&gt;
#&lt;br /&gt;
# when using the Dynamore tools, you can start something like this at the end&lt;br /&gt;
DM.plotcprs.lnx -merge &amp;gt;&amp;gt; dyna.log&lt;br /&gt;
#&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
===LSTC Tools: LS-OPT and LS-PREPOST===&lt;br /&gt;
&lt;br /&gt;
For the new Rocky 9 cluster, I have not looked deeply into the ls-opt and ls-prepost packages that were installed. I noticed though that the LSTC server provided access to much newer versions of both software packages. If you would like to learn more or have a specific version in mind, I can easily download and install it for you.&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
$ module avail ls-opt&lt;br /&gt;
----------------------------------------------- /shared/apps/modulefiles ------------------------------------------------&lt;br /&gt;
ls-opt/5.1.1  ls-opt/6.0.0  ls-opt/7.0.0  ls-opt/7.0.2   ls-opt/2022R2  &lt;br /&gt;
ls-opt/5.2.1  ls-opt/6.1.0  ls-opt/7.0.1  ls-opt/2022R1  ls-opt/2023R1  &lt;br /&gt;
&lt;br /&gt;
$ module avail ls-prepost&lt;br /&gt;
----------------------------------------------- /shared/apps/modulefiles ------------------------------------------------&lt;br /&gt;
ls-prepost/4.5.10  ls-prepost/4.8.13  ls-prepost/4.8.30  ls-prepost/4.9.16  ls-prepost/4.10.7 &lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
===Dynamore Software===&lt;br /&gt;
&lt;br /&gt;
The Dynamore tools are available as a module:&lt;br /&gt;
&lt;br /&gt;
 module load dynamore/current&lt;br /&gt;
&lt;br /&gt;
We typically acquire a yearly license for the tools as we purchase licenses for LS-Dyna.&lt;br /&gt;
&lt;br /&gt;
===Vendor License File Installation===&lt;br /&gt;
&lt;br /&gt;
If you would like for us to install a vendor license for LS-Dyna models, please contact us for the required information. We can send you the general LS-Dyna license file to show the host ids for the license server. Using that information, your vendor should be able to create a vendor license file. Please send that file to us per Email or by other means.&lt;br /&gt;
&lt;br /&gt;
==StarCCM+ on the ARROW Cluster==&lt;br /&gt;
&lt;br /&gt;
===Currently Available StarCCM+ Versions===&lt;br /&gt;
&lt;br /&gt;
As of late 2025, we have the following versions of &#039;&#039;&#039;StarCCM+&#039;&#039;&#039; available on the cluster:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
$ module avail starccm&lt;br /&gt;
---------------------------- /shared/apps/modulefiles ----------------------------&lt;br /&gt;
starccm/15.02.007-R8  starccm/16.06.008-R8  starccm/18.06.006-R8  &lt;br /&gt;
starccm/15.02.009-R8  starccm/17.02.007-R8  starccm/19.02.009-R8  &lt;br /&gt;
starccm/15.04.008-R8  starccm/17.02.008-R8  starccm/20.04.007-R8  &lt;br /&gt;
starccm/15.06.008-R8  starccm/17.04.007-R8  starccm/20.06.007-R8  &lt;br /&gt;
starccm/16.02.008-R8  starccm/17.06.007-R8  &lt;br /&gt;
starccm/16.04.007-R8  starccm/18.04.008-R8&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
If using a &#039;&#039;&#039;single node&#039;&#039;&#039; for StarCCM+, job submission (for an interactive job) is simple and will use appropriate default settings:&lt;br /&gt;
&lt;br /&gt;
 qsub -I -X -q epyc1 -l walltime=20:00:00&lt;br /&gt;
&lt;br /&gt;
StarCCM+ can make use of the job scheduler attributes by automatically obtaining the number of cores and other resources from OpenPBS. In this case, the default number of cores and mpi processes for StarCCM+ are both 64 when using the epyc1 queue. So you can start your StarCCM+ run with:&lt;br /&gt;
&lt;br /&gt;
 module load starccm/15.02.007-R8 (or any other version)&lt;br /&gt;
 starccm+ -bs pbs&lt;br /&gt;
&lt;br /&gt;
In this case, there is no need for StarCCM+ to be told to run the case in parallel with the selected number of cores/mpiprocs.&lt;br /&gt;
&lt;br /&gt;
This can get a bit more complex when running on multiple nodes or when requesting high memory nodes. In that case you would use job submission parameters as shown below:&lt;br /&gt;
&lt;br /&gt;
 qsub -I -X -q epyc1 -l walltime=20:00:00,select=2:ncpus=64:mpiprocs=61:mem=500GB&lt;br /&gt;
&lt;br /&gt;
Requesting nodes that can satisfy those resources, two nodes with these attributes must exist. We have multiple nodes with 512GB in the epyc1 queue, meaning that this job will run on two machines that have at least the required amount of memory installed (on each node). The job will be queued until two machines like this will be available. If no machines with these resources exist, the job will stay in the queue forever. Therefore, you have to craft the submission string carefully.&lt;br /&gt;
&lt;br /&gt;
To accommodate high memory jobs, the nodes have been assigned priorities for assignment. Low memory jobs have the highest priority and will be assigned to nodes that can accommodate the request. High memory nodes have the lowest priority, meaning that they are the last ones given out to users. This makes it more likely that a high memory job can be run soon when the cluster is moderately loaded with jobs.&lt;br /&gt;
&lt;br /&gt;
StarCCM+ will always use the Intel MPI fabric. Other MPI versions do not work, even when selected on the command line.&lt;br /&gt;
&lt;br /&gt;
==OpenFOAM on the ARROW Cluster==&lt;br /&gt;
&lt;br /&gt;
===Currently Available OpenFOAM  Versions===&lt;br /&gt;
&lt;br /&gt;
As of late 2025, we have the following versions of OpenFOAM available on the cluster:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
$ module avail openfoam&lt;br /&gt;
------------ /shared/apps/modulefiles ------------&lt;br /&gt;
openfoam/9   openfoam/13      openfoam/v2312  &lt;br /&gt;
openfoam/10  openfoam/13-amd  openfoam/v2406  &lt;br /&gt;
openfoam/11  openfoam/v2212   &lt;br /&gt;
openfoam/12  openfoam/v2306&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
Contact us if you encounter problems; there can be various reasons why OpenFOAM may have trouble on certain hardware or when compiling dynamic code. When loading OpenFOAM modules, a number of dependencies will be automatically loaded for you, and you don&#039;t have to load those yourself. For example:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
$ module load openfoam/13&lt;br /&gt;
Loading openfoam/13&lt;br /&gt;
  Loading requirement: intel/2024.2.0/mpi/2021.13 gcc/gcc-12.1.0&lt;br /&gt;
&lt;br /&gt;
$ module list&lt;br /&gt;
Currently Loaded Modulefiles:&lt;br /&gt;
 1) intel/2024.2.0/mpi/2021.13   2) gcc/gcc-12.1.0   3) openfoam/13&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
In this case, OpenFOAM 13 loads the Intel 2024 MPI module, and loads the GCC compiler 12.1. OpenFOAM was compiled from source, and has been compiled specifically with that compiler and MPI version, so it make little sense to use other compilers or MPI libraries.&lt;br /&gt;
&lt;br /&gt;
Note: We have found a problem with running the Intel 2024 MPI library in the amd64 queue. Therefore, we have a modified module that uses the Intel 2022 library (I know -- Intel 2022 gives you the 2021 MPI libraries, but that is the way Intel distributes this software):&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
$ module load openfoam/13-amd &lt;br /&gt;
Loading mpi version 2021.7.0&lt;br /&gt;
Loading openfoam/13-amd&lt;br /&gt;
  Loading requirement: intel/2022.2.0/mpi/2021.7.0 gcc/gcc-12.1.0&lt;br /&gt;
&lt;br /&gt;
$ module list&lt;br /&gt;
Currently Loaded Modulefiles:&lt;br /&gt;
 1) intel/2022.2.0/mpi/2021.7.0   2) gcc/gcc-12.1.0   3) openfoam/13-amd&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
If you are compiling OpenFOAM yourself, the modules are of little help. You would need to select the appropriate MPI version and compiler before doing so, and then consistently load them before running your OpenFOAM executables. Within the &amp;quot;etc/bashrc&amp;quot; file in the source code tree, you want to set the MPI library to INTELMPI. As usual with self-compiled versions of OpenFOAM, you would &amp;quot;source etc/bashrc&amp;quot; to set up your personal environment to run your home-brew version of OpenFOAM. Contact us if you need to learn more about compiling OpenFOAM on the system.&lt;br /&gt;
&lt;br /&gt;
==Additional Software Applications and Libraries==&lt;br /&gt;
&lt;br /&gt;
===Loadable GCC Compiler Versions===&lt;br /&gt;
&lt;br /&gt;
The Rocky 9.6 operating system uses the GCC 11.5 compiler. That should be sufficient for most users when compiling your own applications. In case you need to use either a more up-to-date compiler, or if you need an older compiler for compatibility, we make the following versions available as loadable modules.&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
$ module avail gcc&lt;br /&gt;
------------ /shared/apps/modulefiles ------------&lt;br /&gt;
gcc/gcc-4.9.4  gcc/gcc-7.5.0  gcc/gcc-10.3.0  &lt;br /&gt;
gcc/gcc-5.5.0  gcc/gcc-8.5.0  gcc/gcc-11.3.0  &lt;br /&gt;
gcc/gcc-6.5.0  gcc/gcc-9.5.0  gcc/gcc-12.1.0&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
Additional versions can be created and made available as modules as well. If you need a specific version that is not currently available, please ask us to compiler and install it. If necessary, we may be able to provide access to other compilers, for example LLVM. We do not provide access to proprietary compilers at this time.&lt;br /&gt;
&lt;br /&gt;
===MPI Libraries and Runtimes===&lt;br /&gt;
&lt;br /&gt;
While we seem to have a variety of MPI versions and flavors available to users, the only MPI versions that allow us to run software over Infiniband are the Intel MPI libraries. Some of the installed alternatives are likely to fail, or will have a set of environment variables that have to be set. All major engineering software packages that we offer are pre-configured with specific MPI versions and settings that have been tested and/or provided by the vendors.&lt;br /&gt;
&lt;br /&gt;
Note: Some MPI libraries may seem to work. They may allow your MPI application to run. But inter-process network communication may travel through the rather slow and high-latency Ethernet fabric, making MPI applications very ineffective and are probably not worth while.&lt;br /&gt;
&lt;br /&gt;
===MatLab Runtimes===&lt;br /&gt;
&lt;br /&gt;
We can install MatLAB run time libraries as needed and have them available as loadable modules. Recently, we had a problem with MatLAB run time libraries being identified as security vulnerabilities. Contact us if you need them installed for one of your projects.&lt;br /&gt;
&lt;br /&gt;
===Anaconda and variants (miniconda etc)===&lt;br /&gt;
&lt;br /&gt;
Our current practice is to have users download and install their own versions of Anaconda and its variants in their own home directories. This allows for maximum flexibility when it comes to installable software modules, and users can maintain the installation, upgrades, and maintenance themselves. If you encounter issues, please contact us. One known side effect of Anaconda installations is a performance hit when starting your software, e.g. python scripts may take 30 seconds or more to execute. This is an artefact caused by the Lustre file system, which has been designed for large files accessible from many machines simultaneously. Performance on reading many small files has not been considered and is fairly poor. Again, contact us and we will design a solution for you as needed.&lt;/div&gt;</summary>
		<author><name>Ley</name></author>
	</entry>
	<entry>
		<id>https://wiki.anl.gov/wiki_tracc/index.php?title=Job_Submission_and_Monitoring&amp;diff=2488</id>
		<title>Job Submission and Monitoring</title>
		<link rel="alternate" type="text/html" href="https://wiki.anl.gov/wiki_tracc/index.php?title=Job_Submission_and_Monitoring&amp;diff=2488"/>
		<updated>2026-02-23T17:18:19Z</updated>

		<summary type="html">&lt;p&gt;Ley: /* Submitting an LS-Dyna Job */&lt;/p&gt;
&lt;hr /&gt;
&lt;div&gt;{| align=&amp;quot;right&amp;quot;&lt;br /&gt;
| __TOC__&lt;br /&gt;
|}&lt;br /&gt;
==Resource Summary View==&lt;br /&gt;
&lt;br /&gt;
To get started, users can query the overall status of resources on the cluster. The &#039;&#039;&#039;&amp;quot;qsum&amp;quot;&#039;&#039;&#039; script will list all queues and nodes, as well as how many are offline, down, free, or assigned to users. This is a script developed by our team, and may need to be updated if something goes wrong. Please contact us if you experience any problems.&lt;br /&gt;
&lt;br /&gt;
Each queue groups a number of nodes together based on their hardware and software configurations. Nodes can be part of more than one queue, and there are other complex details that we are ignoring here for the purpose of keeping it simple.&lt;br /&gt;
&lt;br /&gt;
===Queues===&lt;br /&gt;
&lt;br /&gt;
Here is a very brief summary of what each of the queues is, and how to use them efficiently:&lt;br /&gt;
&lt;br /&gt;
; a4000: This is a queue that has three 16-core CPU machines, each of which is furthermore equipped with three A4000 GPUs. That makes a total of 9 A4000 GPUs available to users. Neither the GPUs nor the processors are particularly powerful these days. The machines have 500GB of memory though, which makes for a good platform for experimenting with GPU capabilities.&lt;br /&gt;
; a6000: This is a queue that has only one 64-core CPU machines, and is equipped with four A6000 GPUs. The system can be upgraded to 8 A6000 GPUs if needed. This is a decent GPU machine that can take a solid workload these days. The machine has 750GB of memory, which makes for a good production platform.&lt;br /&gt;
; amd16: This is a queue with many of our older AMD-based 16-core machines, each of which has 30GB of memory. While individual machines are a bit outdated, they are all interconnected with Infiniband and can provide a solid production workload in multi-nodes jobs over MPI without blocking the more current (and thus expensive) systems.&lt;br /&gt;
; epyc1/epyc2: These are 2 separate queues with slightly different performance characteristics. Each of the groups is interconnected with Infiniband to provide a platform for large and demanding software packages, such as LS-Dyna and StarCCM+. They have between 250GB and 500GB of memory. Because licenses for these software packages are very expensive, they should use these two queues for making optimum use of limited core licenses available to each package.&lt;br /&gt;
; xeon28: This is a set of intermediate machines with 28 cores and 64GB of memory. They can be used for a variety of purposes, including MPI jobs and single node application software.&lt;br /&gt;
; virtual: This is a set of nodes without MPI capabilities. They are virtual machines with 32GB each. They can be used for higher demand applications that would interfere with the login nodes, and therefore with other users of these login machines. A user would submit interactive jobs to individual virtual machines and avoid any significant load on login nodes.&lt;br /&gt;
&lt;br /&gt;
===The Queue Summary Script (qsum)===&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
$ qsum&lt;br /&gt;
=============== a4000 ==========================================================&lt;br /&gt;
Queue: &amp;quot;a4000&amp;quot; / nodes: 3 / down: 0 / offline: 0 / busy: 0 / available: 3&lt;br /&gt;
      AVAILABLE (3): g001, g002, g003&lt;br /&gt;
=============== a6000 ==========================================================&lt;br /&gt;
Queue: &amp;quot;a6000&amp;quot; / nodes: 1 / down: 0 / offline: 0 / busy: 0 / available: 1&lt;br /&gt;
      AVAILABLE (1): lambda01&lt;br /&gt;
=============== amd16 ==========================================================&lt;br /&gt;
Queue: &amp;quot;amd16&amp;quot; / nodes: 33 / down: 2 / offline: 0 / busy: 2 / available: 29&lt;br /&gt;
           DOWN (2): n017, n030&lt;br /&gt;
            ley (2): n001, n002&lt;br /&gt;
     AVAILABLE (29): n003, n004, n005, n006, n007, n008, n009, n010, n011, n012&lt;br /&gt;
                     n013, n014, n015, n016, n018, n019, n020, n021, n022, n023&lt;br /&gt;
                     n024, n025, n026, n027, n028, n029, n031, n032, n039&lt;br /&gt;
=============== epyc1 ==========================================================&lt;br /&gt;
Queue: &amp;quot;epyc1&amp;quot; / nodes: 1 / down: 0 / offline: 0 / busy: 0 / available: 1&lt;br /&gt;
      AVAILABLE (1): a027&lt;br /&gt;
=============== epyc2 ==========================================================&lt;br /&gt;
Queue: &amp;quot;epyc2&amp;quot; / nodes: 20 / down: 0 / offline: 0 / busy: 5 / available: 15&lt;br /&gt;
            ley (2): a030, a031&lt;br /&gt;
         msitek (3): a028, a029, a032&lt;br /&gt;
     AVAILABLE (15): a033, a034, a035, a036, a037, a038, a039, a040, a041, a042&lt;br /&gt;
                     a043, a044, a045, a046, a047&lt;br /&gt;
=============== virtual ========================================================&lt;br /&gt;
Queue: &amp;quot;virtual&amp;quot; / nodes: 6 / down: 0 / offline: 0 / busy: 0 / available: 6&lt;br /&gt;
      AVAILABLE (6): v001, v002, v003, v004, v005, v006&lt;br /&gt;
=============== xeon28 =========================================================&lt;br /&gt;
Queue: &amp;quot;xeon28&amp;quot; / nodes: 12 / down: 0 / offline: 0 / busy: 0 / available: 12&lt;br /&gt;
     AVAILABLE (12): p001, p002, p003, p004, p005, p006, p007, p008, p009, p010&lt;br /&gt;
                     p011, p012&lt;br /&gt;
================================================================================&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
==Queue Status and Monitoring Jobs==&lt;br /&gt;
&lt;br /&gt;
===qstat===&lt;br /&gt;
&lt;br /&gt;
To find out out about the status of all running jobs on the cluster you can use the &#039;&#039;&#039;&amp;quot;qstat&amp;quot;&#039;&#039;&#039; command. Here is an example:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
$ qstat&lt;br /&gt;
&lt;br /&gt;
Nov 20 18:30 ley@login3:Plots$ qstat&lt;br /&gt;
Job id            Name             User              Time Use S Queue&lt;br /&gt;
----------------  ---------------- ----------------  -------- - -----&lt;br /&gt;
3023.pbs          STDIN            msitek            4144:14* R epyc2           &lt;br /&gt;
3029.pbs          STDIN            ley               76:46:53 R epyc2           &lt;br /&gt;
3032.pbs          STDIN            msitek            2879:52* R epyc2           &lt;br /&gt;
3033.pbs          STDIN            msitek            3687:29* R epyc2           &lt;br /&gt;
3048.pbs          foo.sh           james.cook               0 Q amd16           &lt;br /&gt;
3060.pbs          of13.sh          ley               310:47:* R epyc2           &lt;br /&gt;
3061.pbs          of13.sh          ley               308:37:* R epyc2           &lt;br /&gt;
3062.pbs          of13.sh          ley               308:02:* R epyc2           &lt;br /&gt;
3063.pbs          of13.sh          ley               308:15:* R epyc2&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
The first column shows the &#039;&#039;&#039;job id&#039;&#039;&#039;, a unique identifier for all jobs ever submitted to the cluster. This job id is important when killing jobs, or for other actions you may need to take.&lt;br /&gt;
&lt;br /&gt;
The next column shows the name of the job script. If the column shows &#039;&#039;&#039;STDIN&#039;&#039;&#039;, it means that this is an &#039;&#039;&#039;interactive job&#039;&#039;&#039; where a user can enter commands in a terminal window. This is particularly useful for model and software development task where the application has to be started and killed repeatedly.&lt;br /&gt;
&lt;br /&gt;
The owner of the job is shown next. These are the user names of the various people using the cluster.&lt;br /&gt;
&lt;br /&gt;
The last three columns indicate the current run time of the job, whether it is running (R) or waiting (Q) for execution. The last entry shows the queue in which the job is running.&lt;br /&gt;
&lt;br /&gt;
===qstat -an1===&lt;br /&gt;
&lt;br /&gt;
Adding a few options gives much more detail about each jobs.&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
qstat -an1&lt;br /&gt;
&lt;br /&gt;
Nov 20 13:09 ley@login3:Plots$ qstat -an1&lt;br /&gt;
&lt;br /&gt;
pbs: &lt;br /&gt;
                                                            Req&#039;d  Req&#039;d   Elap&lt;br /&gt;
Job ID          Username Queue    Jobname    SessID NDS TSK Memory Time  S Time&lt;br /&gt;
--------------- -------- -------- ---------- ------ --- --- ------ ----- - -----&lt;br /&gt;
3023.pbs        msitek   epyc2    STDIN      24360*   1  64  350gb 100:0 R 81:46 a028/0*64&lt;br /&gt;
3029.pbs        ley      epyc2    STDIN      21719*   2 128  100gb 200:0 R 72:31 a030/0*64+a031/0*64&lt;br /&gt;
3032.pbs        msitek   epyc2    STDIN      18102*   1  64  350gb 100:0 R 57:57 a029/0*64&lt;br /&gt;
3033.pbs        msitek   epyc2    STDIN      830486   1  64  350gb 100:0 R 57:53 a032/0*64&lt;br /&gt;
3048.pbs        james.c* amd16    foo.sh        --    1  28   30gb 01:00 Q   --   -- &lt;br /&gt;
3060.pbs        ley      epyc2    STDIN      763101   1  64  350gb 48:00 R 06:42 a033/0*64&lt;br /&gt;
3061.pbs        ley      epyc2    STDIN      763947   1  64  350gb 48:00 R 06:40 a034/0*64&lt;br /&gt;
3062.pbs        ley      epyc2    STDIN      761473   1  64  350gb 48:00 R 06:39 a035/0*64&lt;br /&gt;
3063.pbs        ley      epyc2    STDIN      766205   1  64  350gb 48:00 R 06:40 a036/0*64&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
In this table, you can see how many nodes and cores are being used by each job. For example, job 3029 of the user &amp;quot;ley&amp;quot; shows that it is running on 2 nodes using a total of 128 cores. In addition to the elapsed time, the table also show the reserved time for this job. This allow you to estimate when a job will be definitely finalized (or killed by the system if still running).&lt;br /&gt;
&lt;br /&gt;
The last column (without a header) is written because the option &#039;&#039;&#039;&amp;quot;-an1&amp;quot;&#039;&#039;&#039; was used. This useful to learn about which nodes are used by each job.&lt;br /&gt;
&lt;br /&gt;
===qstat -q===&lt;br /&gt;
&lt;br /&gt;
To learn more about the queues on the cluster, the &#039;&#039;&#039;&amp;quot;-q&amp;quot;&#039;&#039;&#039; option turns out to be useful.&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
$ qstat -q&lt;br /&gt;
&lt;br /&gt;
server: pbs&lt;br /&gt;
&lt;br /&gt;
Queue            Memory CPU Time Walltime Node   Run   Que   Lm  State&lt;br /&gt;
---------------- ------ -------- -------- ---- ----- ----- ----  -----&lt;br /&gt;
virtual            30gb    --       --       1     0     0   --   E R&lt;br /&gt;
a4000             500gb    --       --       1     0     0   --   E R&lt;br /&gt;
a6000             750gb    --       --       1     0     0   --   E R&lt;br /&gt;
xeon28             --      --       --       4     0     0   --   E R&lt;br /&gt;
amd16              --      --       --       8     0     1   --   E R&lt;br /&gt;
epyc2              --      --       --       2    14     0   --   E R&lt;br /&gt;
epyc1              --      --       --       2     0     0   --   E R&lt;br /&gt;
                                               ----- -----&lt;br /&gt;
                                                  14     1&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
For each queue, some basic values are displayed. The first three queues listed above have a default memory allocation as shown, and the column &#039;&#039;&#039;&amp;quot;Node&amp;quot;&#039;&#039;&#039; indicates the maximum number of nodes that can be asked for at job submission time. For example, you can request just one node for a job from the first three queues (because these are queues where MPI makes no sense). The &#039;&#039;&#039;&amp;quot;xeon28&amp;quot;&#039;&#039;&#039; queue also for a maximum of 4 nodes per MPI job. The &#039;&#039;&#039;&amp;quot;amd16&amp;quot;&#039;&#039;&#039; queue has a maximum of 8 nodes per job, and the &#039;&#039;&#039;&amp;quot;epyc1&amp;quot;&#039;&#039;&#039; and &#039;&#039;&#039;&amp;quot;epyc2&amp;quot;&#039;&#039;&#039; queues have maxima of two nodes per job. These limitations can be changed by the administrator as needed. As shown above, this will prevent inefficient resource requests.&lt;br /&gt;
&lt;br /&gt;
===qstat -f===&lt;br /&gt;
&lt;br /&gt;
The command &#039;&#039;&#039;&amp;quot;qstat -f -F json 3029&amp;quot;&#039;&#039;&#039; retrieves extremely detailed stats on the running job 3029. The result can be returned in JSON format to be ready for further processing (shown below).&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
$ qstat -f -F json 3029&lt;br /&gt;
{&lt;br /&gt;
    &amp;quot;timestamp&amp;quot;:1763705353,&lt;br /&gt;
    &amp;quot;pbs_version&amp;quot;:&amp;quot;23.06.06&amp;quot;,&lt;br /&gt;
    &amp;quot;pbs_server&amp;quot;:&amp;quot;pbs&amp;quot;,&lt;br /&gt;
    &amp;quot;Jobs&amp;quot;:{&lt;br /&gt;
        &amp;quot;3029.pbs&amp;quot;:{&lt;br /&gt;
            &amp;quot;Job_Name&amp;quot;:&amp;quot;STDIN&amp;quot;,&lt;br /&gt;
            &amp;quot;Job_Owner&amp;quot;:&amp;quot;ley@login4&amp;quot;,&lt;br /&gt;
            &amp;quot;resources_used&amp;quot;:{&lt;br /&gt;
                &amp;quot;cpupercent&amp;quot;:98,&lt;br /&gt;
                &amp;quot;cput&amp;quot;:&amp;quot;76:46:53&amp;quot;,&lt;br /&gt;
                &amp;quot;hpmem&amp;quot;:&amp;quot;0b&amp;quot;,&lt;br /&gt;
                &amp;quot;mem&amp;quot;:&amp;quot;52428800kb&amp;quot;,&lt;br /&gt;
                &amp;quot;ncpus&amp;quot;:128,&lt;br /&gt;
                &amp;quot;vmem&amp;quot;:&amp;quot;52428800kb&amp;quot;,&lt;br /&gt;
                &amp;quot;walltime&amp;quot;:&amp;quot;78:09:32&amp;quot;&lt;br /&gt;
            },&lt;br /&gt;
            &amp;quot;job_state&amp;quot;:&amp;quot;R&amp;quot;,&lt;br /&gt;
            &amp;quot;queue&amp;quot;:&amp;quot;epyc2&amp;quot;,&lt;br /&gt;
            &amp;quot;server&amp;quot;:&amp;quot;pbs&amp;quot;,&lt;br /&gt;
            &amp;quot;Checkpoint&amp;quot;:&amp;quot;u&amp;quot;,&lt;br /&gt;
            &amp;quot;ctime&amp;quot;:&amp;quot;Mon Nov 17 17:58:25 2025&amp;quot;,&lt;br /&gt;
            &amp;quot;Error_Path&amp;quot;:&amp;quot;/dev/pts/0&amp;quot;,&lt;br /&gt;
            &amp;quot;exec_host&amp;quot;:&amp;quot;a030/0*64+a031/0*64&amp;quot;,&lt;br /&gt;
            &amp;quot;exec_vnode&amp;quot;:&amp;quot;(a030:ncpus=64:mem=52428800kb)+(a031:ncpus=64:mem=52428800kb)&amp;quot;,&lt;br /&gt;
            &amp;quot;Hold_Types&amp;quot;:&amp;quot;n&amp;quot;,&lt;br /&gt;
            &amp;quot;interactive&amp;quot;:&amp;quot;True&amp;quot;,&lt;br /&gt;
            &amp;quot;Join_Path&amp;quot;:&amp;quot;n&amp;quot;,&lt;br /&gt;
            &amp;quot;Keep_Files&amp;quot;:&amp;quot;n&amp;quot;,&lt;br /&gt;
            &amp;quot;Mail_Points&amp;quot;:&amp;quot;a&amp;quot;,&lt;br /&gt;
            &amp;quot;mtime&amp;quot;:&amp;quot;Fri Nov 21 00:07:59 2025&amp;quot;,&lt;br /&gt;
            &amp;quot;Output_Path&amp;quot;:&amp;quot;/dev/pts/0&amp;quot;,&lt;br /&gt;
            &amp;quot;Priority&amp;quot;:0,&lt;br /&gt;
            &amp;quot;qtime&amp;quot;:&amp;quot;Mon Nov 17 17:58:25 2025&amp;quot;,&lt;br /&gt;
            &amp;quot;Rerunable&amp;quot;:&amp;quot;False&amp;quot;,&lt;br /&gt;
            &amp;quot;Resource_List&amp;quot;:{&lt;br /&gt;
                &amp;quot;mem&amp;quot;:&amp;quot;100gb&amp;quot;,&lt;br /&gt;
                &amp;quot;mpiprocs&amp;quot;:128,&lt;br /&gt;
                &amp;quot;ncpus&amp;quot;:128,&lt;br /&gt;
                &amp;quot;nodect&amp;quot;:2,&lt;br /&gt;
                &amp;quot;place&amp;quot;:&amp;quot;free&amp;quot;,&lt;br /&gt;
                &amp;quot;select&amp;quot;:&amp;quot;2:ncpus=64:mem=50gb:mpiprocs=64&amp;quot;,&lt;br /&gt;
                &amp;quot;walltime&amp;quot;:&amp;quot;200:00:00&amp;quot;&lt;br /&gt;
            },&lt;br /&gt;
            &amp;quot;stime&amp;quot;:&amp;quot;Mon Nov 17 17:58:25 2025&amp;quot;,&lt;br /&gt;
            &amp;quot;session_id&amp;quot;:2171964,&lt;br /&gt;
            &amp;quot;jobdir&amp;quot;:&amp;quot;/mnt/lustre/arrow/home/ley&amp;quot;,&lt;br /&gt;
            &amp;quot;substate&amp;quot;:42,&lt;br /&gt;
            &amp;quot;Variable_List&amp;quot;:{&lt;br /&gt;
                &amp;quot;PBS_O_HOME&amp;quot;:&amp;quot;/mnt/lustre/arrow/home/ley&amp;quot;,&lt;br /&gt;
                &amp;quot;PBS_O_LANG&amp;quot;:&amp;quot;en_US.UTF-8&amp;quot;,&lt;br /&gt;
                &amp;quot;PBS_O_LOGNAME&amp;quot;:&amp;quot;ley&amp;quot;,&lt;br /&gt;
                &amp;quot;PBS_O_PATH&amp;quot;:&amp;quot;/shared/apps/active/lstc/lsprepost/SP-4.5:/shared/apps/active/lstc/lsprepost/DP-4.3:/shared/bin:/usr/share/Modules/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/pbs/bin:/opt/thinlinc/bin:/opt/thinlinc/sbin:/mnt/lustre/arrow/home/ley/.local/bin:/mnt/lustre/arrow/home/ley/bin&amp;quot;,&lt;br /&gt;
                &amp;quot;PBS_O_MAIL&amp;quot;:&amp;quot;/var/spool/mail/ley&amp;quot;,&lt;br /&gt;
                &amp;quot;PBS_O_SHELL&amp;quot;:&amp;quot;/bin/bash&amp;quot;,&lt;br /&gt;
                &amp;quot;PBS_O_WORKDIR&amp;quot;:&amp;quot;/mnt/lustre/arrow/home/ley/Qualification/LS-Dyna/Rocky9/Seatbelt/Template&amp;quot;,&lt;br /&gt;
                &amp;quot;PBS_O_SYSTEM&amp;quot;:&amp;quot;Linux&amp;quot;,&lt;br /&gt;
                &amp;quot;PBS_O_QUEUE&amp;quot;:&amp;quot;epyc2&amp;quot;,&lt;br /&gt;
                &amp;quot;PBS_O_HOST&amp;quot;:&amp;quot;login4&amp;quot;&lt;br /&gt;
            },&lt;br /&gt;
            &amp;quot;comment&amp;quot;:&amp;quot;Job run at Mon Nov 17 at 17:58 on (a030:ncpus=64:mem=52428800kb)+(a031:ncpus=64:mem=52428800kb)&amp;quot;,&lt;br /&gt;
            &amp;quot;etime&amp;quot;:&amp;quot;Mon Nov 17 17:58:25 2025&amp;quot;,&lt;br /&gt;
            &amp;quot;run_count&amp;quot;:1,&lt;br /&gt;
            &amp;quot;Submit_arguments&amp;quot;:&amp;quot;-I -q epyc2 -l walltime=200:00:00,select=2:ncpus=64:mem=50gb:mpiprocs=64&amp;quot;,&lt;br /&gt;
            &amp;quot;project&amp;quot;:&amp;quot;_pbs_project_default&amp;quot;,&lt;br /&gt;
            &amp;quot;Submit_Host&amp;quot;:&amp;quot;login4&amp;quot;&lt;br /&gt;
        }&lt;br /&gt;
    }&lt;br /&gt;
}&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
===Manual pages for qstat===&lt;br /&gt;
&lt;br /&gt;
To learn more about the &#039;&#039;&#039;&amp;quot;qstat&amp;quot;&#039;&#039;&#039; command, you can use the command &#039;&#039;&#039;&amp;quot;man qstat&amp;quot;&#039;&#039;&#039;, which will print a lot of detailed information about the capabilities of this command.&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
$ man qstat&lt;br /&gt;
&lt;br /&gt;
qstat(1B)                                       PBS Professional                                       qstat(1B)&lt;br /&gt;
&lt;br /&gt;
NAME&lt;br /&gt;
       qstat - display status of PBS jobs, queues, or servers&lt;br /&gt;
&lt;br /&gt;
SYNOPSIS&lt;br /&gt;
       Displaying Job Status&lt;br /&gt;
              Default format:&lt;br /&gt;
              qstat [-E] [-J] [-p] [-t] [-w] [-x] [[&amp;lt;job ID&amp;gt; | &amp;lt;destination&amp;gt;] ...]&lt;br /&gt;
&lt;br /&gt;
              Long format:&lt;br /&gt;
              qstat -f [-F json | dsv [-D &amp;lt;delimiter&amp;gt;]] [-E] [-J] [-p] [-t] [-w]&lt;br /&gt;
                    [-x] [[&amp;lt;job ID&amp;gt; | &amp;lt;destination&amp;gt;] ...]&lt;br /&gt;
... &amp;lt;many more pages&amp;gt;&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
==Job Submission Basics==&lt;br /&gt;
&lt;br /&gt;
Jobs are submitted into the system using the &#039;&#039;&#039;&amp;quot;qsub&amp;quot;&#039;&#039;&#039; application. This application can take many different options and allows for a lot of different resource requests to tell the cluster what to do. We are running &#039;&#039;&#039;OpenPBS 23.06.06&#039;&#039;&#039; as our job scheduler. Here is a link to the User&#039;s Manual (of PBS PRO) if you want to explore gory details and capabilities. The User&#039;s Guide has about 240 pages, the Reference Guide has 500 pages, and the Big Book has 2500 pages. So there is a lot of information available. I also added job submission info for the LCRC cluster.&lt;br /&gt;
&lt;br /&gt;
* [https://argonne-lcrc.github.io/user-guides/running-jobs-at-lcrc/pbs-pro/ Argonne&#039;s LCRC pages on job submissions on their clusters]&lt;br /&gt;
* [https://help.altair.com/2022.1.0/PBS%20Professional/PBSUserGuide2022.1.pdf PBS Professional 2022.1 User&#039;s Guide]&lt;br /&gt;
* [https://help.altair.com/2022.1.0/PBS%20Professional/PBSReferenceGuide2022.1.pdf PBS Professional 2022.1 Reference Guide]&lt;br /&gt;
* [https://help.altair.com/2022.1.0/PBS%20Professional/PBS2022.1.pdf Altair PBS Professional 2022.1 Big Book]&lt;br /&gt;
&lt;br /&gt;
The User&#039;s Guide can be very helpful to clarify some of the concepts and capabilities, but it can be hard to find the specific information you may be looking for. Please understand that we are no longer running TORQUE and MAUI, so the syntax for job submission is distinctively different yet quite similar.&lt;br /&gt;
&lt;br /&gt;
The reference guide may be helpful to understand the complete syntax and full capabilities of the software.&lt;br /&gt;
&lt;br /&gt;
The big book is what I had to use when configuring OpenPBS earlier this year. This includes all the tricky details needed to make the system work smoothly for us. It&#039;s a bit scary to look at a PDF file that is 2500 pages long, but that is nothing compared to the StarCCM+ manuals.&lt;br /&gt;
&lt;br /&gt;
&amp;lt;BLOCKQUOTE&amp;gt;&lt;br /&gt;
[[File:Attention.jpg|25px]] &#039;&#039;&#039;IMPORTANT NOTE:&#039;&#039;&#039; &#039;&#039;The following sections are important to understand. They explain how jobs are submitted and then scjeduled for execution based on resources available and the specific need of the user.&#039;&#039;&lt;br /&gt;
&amp;lt;/BLOCKQUOTE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
The following sections explain the various tasks you may want to submit fir execution, ordered from simple to complex.&lt;br /&gt;
&lt;br /&gt;
* General Batch Jobs&lt;br /&gt;
** Requesting a Single Node for a Job&lt;br /&gt;
** Requesting Multiple Nodes for a Job&lt;br /&gt;
* Embedded Job Resource Requests&lt;br /&gt;
* Interactive Jobs&lt;br /&gt;
* Interactive Jobs with X-Windows GUI Applications&lt;br /&gt;
* Running Multiple Jobs on Single Nodes&lt;br /&gt;
* Running Jobs using GPUs&lt;br /&gt;
&lt;br /&gt;
===General Batch Jobs===&lt;br /&gt;
&lt;br /&gt;
Let&#039;s get started with a very basic usage of the system. Let&#039;s assume you have a simple application, and you want to execute it on a cluster node. Let&#039;s also assume that this is a very simple application, one that runs on one or a few cores, doesn&#039;t require any keyboard interaction with the user, doesn&#039;t need the user to see what&#039;s typically written to the screen, and writes its output to files. In this case, we can submit this application as a batch job, which will place it into an execution queue and process it as soon as a node becomes available.&lt;br /&gt;
&lt;br /&gt;
If the application requires more cores than a single node can provide, we can run the application over Infiniband with MPI message passing. In this case, we need to understand the concept of MPI applications a bit better. In both cases, we get started by creating a folder on the file system. Naming conventions are important so that you can distinguish the jobs by folder name.&lt;br /&gt;
&lt;br /&gt;
For both of the above scenarios, you would typically create a Bash shell script, and then submit the script into one of the queues for eventual execution.&lt;br /&gt;
&lt;br /&gt;
====Requesting a Single Node for a Job====&lt;br /&gt;
&lt;br /&gt;
Let&#039;s try something rather trivial to get used to the concept. Create yourself a folder, for example &#039;&#039;&#039;&amp;quot;myjobfolder&amp;quot;&#039;&#039;&#039;. Within that folder, create a job submission script. That script can have any name, but something short and simple may be best. Let&#039;s assume you create a file called &#039;&#039;&#039;&amp;quot;cluster.job&amp;quot;&#039;&#039;&#039;. The file doesn&#039;t have to have that extension. Any file name will do. But using the same filename for all of your jobs helps finding your way around the many files that will be created over time. The &#039;&#039;&#039;&amp;quot;cluster.job&amp;quot;&#039;&#039;&#039; file should look something like this:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
#!/bin/bash&lt;br /&gt;
#&lt;br /&gt;
# the following ensures that you will change into the directory where you are&lt;br /&gt;
# submitting the job from.&lt;br /&gt;
cd $PBS_O_WORKDIR&lt;br /&gt;
#&lt;br /&gt;
# now we sleep for 60 seconds and waste time. This is a placeholder for your application,&lt;br /&gt;
# which would be doing useful work for you.&lt;br /&gt;
sleep 60&lt;br /&gt;
#&lt;br /&gt;
# and after doing things, we may want to write something into a file to show that&lt;br /&gt;
# our jobs is done.&lt;br /&gt;
echo `date` &amp;gt; info.log&lt;br /&gt;
#&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
This can be submitted without detailed resource specifications (except for the walltime, which is be default 0:00:00):&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
$ qsub -q virtual -l walltime=1:00:00 cluster.job &lt;br /&gt;
3072.pbs&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
Wait a little, then check the status of running jobs:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
$ qstat -an1&lt;br /&gt;
&lt;br /&gt;
pbs: &lt;br /&gt;
                                                            Req&#039;d  Req&#039;d   Elap&lt;br /&gt;
Job ID          Username Queue    Jobname    SessID NDS TSK Memory Time  S Time&lt;br /&gt;
--------------- -------- -------- ---------- ------ --- --- ------ ----- - -----&lt;br /&gt;
3023.pbs        msitek   epyc2    STDIN      24360*   1  64  350gb 100:0 R 83:17 a028/0*64&lt;br /&gt;
3029.pbs        ley      epyc2    STDIN      21719*   2 128  100gb 200:0 R 74:00 a030/0*64+a031/0*64&lt;br /&gt;
3033.pbs        msitek   epyc2    STDIN      830486   1  64  350gb 100:0 R 59:23 a032/0*64&lt;br /&gt;
3048.pbs        james.c* amd16    foo.sh        --    1  28   30gb 01:00 Q   --   -- &lt;br /&gt;
3060.pbs        ley      epyc2    STDIN      763101   1  64  350gb 48:00 R 08:10 a033/0*64&lt;br /&gt;
3061.pbs        ley      epyc2    STDIN      763947   1  64  350gb 48:00 R 08:10 a034/0*64&lt;br /&gt;
3070.pbs        ley      epyc2    STDIN      766847   1  64  350gb 48:00 R 07:23 a042/0*64&lt;br /&gt;
3072.pbs        ley      virtual  cluster.j* 230230   1   4   30gb 01:00 E 00:01 v001/0*4&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
In this particular example, we are sending this job to the &#039;&#039;&#039;queue &amp;quot;virtual&amp;quot;&#039;&#039;&#039;. This queue, by default, allocates 30GB of memory to the job, and runs on 1 node with 4 cores. This is sufficient capacity to run quite a workload. When submitting a job to a single node, &#039;&#039;&#039;reasonable maximum allocations are automatically assigned&#039;&#039;&#039;, and the user doesn&#039;t have to worry about running out of memory or how many cores he will be using.&lt;br /&gt;
&lt;br /&gt;
The only required argument is the &#039;&#039;&#039;&amp;quot;walltime&amp;quot;&#039;&#039;&#039; argument. By default, the job will quit as soon as it is submitted. This indicates to the user that he forgot to provide the &#039;&#039;&#039;&amp;quot;walltime&amp;quot;&#039;&#039;&#039; argument.&lt;br /&gt;
&lt;br /&gt;
When the job disappears from the job list, it is done. At this point, you will find the file &amp;quot;info.log&amp;quot; in your job folder.&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
$ cat info.log &lt;br /&gt;
Thu Nov 20 08:00:31 PM CST 2025&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
====Requesting Multiple Nodes for a Job====&lt;br /&gt;
&lt;br /&gt;
To run jobs on multiple nodes, you will be likely &#039;&#039;&#039;executing jobs using MPI&#039;&#039;&#039;, the message passing interface. This establishes high-speed low-latency interconnections between the cores on one machine and the cores on the other machines. Data transfer does not require involvement of the cores themselves. Instead, the core tell the InfiniBand interconnect (and cores on the same node through shared memory) to transfer the data through RDMA, remoted direct memory access. The cores don&#039;t need to spend CPU cycles on copying data, but rather simply access the data once it has been copied by the Infiniband fabric. This makes for extremely efficient remote memory access, and message passing is used to coordinate data transfer between the cores no matter where they are located on any of the nodes.&lt;br /&gt;
&lt;br /&gt;
On our cluster, MPI-aware applications like &#039;&#039;&#039;OpenFOAM&#039;&#039;&#039;, &#039;&#039;&#039;StarCCM+&#039;&#039;&#039;, and &#039;&#039;&#039;LS-Dyna&#039;&#039;&#039; can be loaded as modules, which then automatically selects the most appropriate MPI library to use. The software applications have been tested to ensure that they work out-of-the box if a user selects any specific version of any of the applications.&lt;br /&gt;
&lt;br /&gt;
The following is a very trivial example for the MPI execution of a very simple executable, with one copy running on each core of the nodes allocated to  the job. It doesn&#039;t perform any real work and just wastes resources for a short time, but it illustrates how execution on the cores of various nodes works.&lt;br /&gt;
&lt;br /&gt;
Like in the previous section, we start with a simple job script that we submit to an appropriate queues. In this case, we pick a queue that has machines with Infiniband interfaces supporting efficient communications. Let&#039;s assume we edit a file with the name &#039;&#039;&#039;&amp;quot;parallel.job&amp;quot;&#039;&#039;&#039; like this:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
#!/bin/bash&lt;br /&gt;
#&lt;br /&gt;
# the following ensures that you will change into the directory where you are&lt;br /&gt;
# submitting the job from.&lt;br /&gt;
cd $PBS_O_WORKDIR&lt;br /&gt;
#&lt;br /&gt;
# to execute a simple command on all of the cores of all of the nodes allocated to the job,&lt;br /&gt;
# we need to make one of the MPI versions available. Let&#039;s use one of the most up-to-date&lt;br /&gt;
# MPI library available on the cluster&lt;br /&gt;
module load intel/2024.2.0/mpi/2021.13&lt;br /&gt;
#&lt;br /&gt;
# now we are apply a few settings that ensure that the MPI library will use the highest-performing&lt;br /&gt;
# Infiniband Interconnect, as well as a few options to tell MPI how to interface nodes with&lt;br /&gt;
# each other and which specific Infiniband adapter to use. This is complex and requires in-depth&lt;br /&gt;
# knowledge of the QLogic Infiniband adapters we are using. It is unlikely that you will ever have to&lt;br /&gt;
# deal with these options, because the &amp;quot;module load&amp;quot; command for the engineering applications we provide&lt;br /&gt;
# on ARROW will handle all those details transparently without the user needing to understand the details.&lt;br /&gt;
export I_MPI_HYDRA_BOOTSTRAP=ssh&lt;br /&gt;
export MPI_DEVICE=rdma:ofa-v2-ib0&lt;br /&gt;
export UCX_NET_DEVICES=qib0:1&lt;br /&gt;
#&lt;br /&gt;
# it doesn&#039;t make much sense, but in this example we are executing the OS command &amp;quot;uptime&amp;quot; on all cores&lt;br /&gt;
# of the nodes allocated to this job. The output from each core is written to the file info.log. We&lt;br /&gt;
# will find 56 lines of output in the file info.log, each created by the corresponding core executing&lt;br /&gt;
# the uptime command.&lt;br /&gt;
mpirun uptime &amp;gt; info.log&lt;br /&gt;
#&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
A good queue to test scripts is the &#039;&#039;&#039;&amp;quot;xeon28&amp;quot;&#039;&#039;&#039; queue. In the queue, we have 2 14-core Xeon processers per node, so that means that each node has 56 actual cores. We do not consider hyperthreading when doing parallel computing. 56 actual cores is what&#039;s being used here. The job submission will look like this:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
qsub -q xeon28 -l walltime=1:0:0 -l select=2:ncpus=28:mpiprocs=28:mem=60G parallel.job&lt;br /&gt;
        ^                  ^               ^       ^           ^      ^ ^ ^&lt;br /&gt;
        |                  |               |       |           |      | | + --- the name of the job script to execute&lt;br /&gt;
        |                  |               |       |           |      | + ----- don&#039;t forget to specify gigabytes&lt;br /&gt;
        |                  |               |       |           |      + ------- the amount of memory to request per node&lt;br /&gt;
        |                  |               |       |           + -------------- the number of MPI tasks per nodes&lt;br /&gt;
        |                  |               |       + -------------------------- the number of cores per node&lt;br /&gt;
        |                  |               + ---------------------------------- the number of nodes to select in the queue&lt;br /&gt;
        |                   + ------------------------------------------------- the requested time, in this case 1h&lt;br /&gt;
        + --------------------------------------------------------------------- the queue to be used for the job&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
At this point, the job has created a file &amp;quot;info.log&amp;quot; with 56 lines, one per core:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
 22:26:05 up 23 days,  9:28,  0 users,  load average: 0.00, 0.00, 0.00&lt;br /&gt;
 22:26:05 up 23 days,  9:28,  0 users,  load average: 0.00, 0.00, 0.00&lt;br /&gt;
 22:26:05 up 23 days,  9:28,  0 users,  load average: 0.00, 0.00, 0.00&lt;br /&gt;
 22:26:05 up 23 days,  9:28,  0 users,  load average: 0.00, 0.00, 0.00&lt;br /&gt;
 22:26:05 up 23 days,  9:28,  0 users,  load average: 0.00, 0.00, 0.00&lt;br /&gt;
 22:26:05 up 23 days,  9:28,  0 users,  load average: 0.00, 0.00, 0.00&lt;br /&gt;
 22:26:05 up 23 days,  9:53,  0 users,  load average: 0.06, 0.03, 0.00&lt;br /&gt;
 22:26:05 up 23 days,  9:53,  0 users,  load average: 0.06, 0.03, 0.00&lt;br /&gt;
 22:26:05 up 23 days,  9:53,  0 users,  load average: 0.06, 0.03, 0.00&lt;br /&gt;
...&lt;br /&gt;
&amp;lt;/PRE&amp;gt; &lt;br /&gt;
&lt;br /&gt;
In this simple example, the lines look all the same. Upon close examination through, you can find slightly different values for some of the lines. Some lines say that the machine is up for 23 days and 9:28, while others say 23 days and 9:53. Because all 28 cores of a node would see the same uptime of the server, half of the entries show one time stamp, and the other 28 cores show the other one. That demonstrates that the 56 processes have been running independently on 2 nodes.&lt;br /&gt;
&lt;br /&gt;
===Embedded Job Resource Requests===&lt;br /&gt;
&lt;br /&gt;
The job script can be modified to embed the resource requests in form of a series of &#039;&#039;&#039;#PBS&#039;&#039;&#039; statements at the beginning of the script file. This is a very common practice use at many HPC installations and job submission engines. Let&#039;s go back to the previous example where we run the script on two nodes in parallel. That is the &#039;&#039;&#039;&amp;quot;parallel.job&amp;quot;&#039;&#039;&#039; script file again:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
#!/bin/bash&lt;br /&gt;
#&lt;br /&gt;
#PBS -q xeon28&lt;br /&gt;
#PBS -l walltime=1:0:0&lt;br /&gt;
#PBS -l select=2:ncpus=28:mpiprocs=28:mem=60G&lt;br /&gt;
#&lt;br /&gt;
# the following ensures that you will change into the directory where you are&lt;br /&gt;
# submitting the job from.&lt;br /&gt;
cd $PBS_O_WORKDIR&lt;br /&gt;
#&lt;br /&gt;
# to execute a simple command on all of the cores of all of the nodes allocated to the job,&lt;br /&gt;
# we need to make one of the MPI versions available. Let&#039;s use one of the most up-to-date&lt;br /&gt;
# MPI library available on the cluster&lt;br /&gt;
module load intel/2024.2.0/mpi/2021.13&lt;br /&gt;
#&lt;br /&gt;
# now we are apply a few settings that ensure that the MPI library will use the highest-performing&lt;br /&gt;
# Infiniband Interconnect, as well as a few options to tell MPI how to interface nodes with&lt;br /&gt;
# each other and which specific Infiniband adapter to use. This is complex and requires in-depth&lt;br /&gt;
# knowledge of the QLogic Infiniband adapters we are using. It is unlikely that you will ever have to&lt;br /&gt;
# deal with these options, because the &amp;quot;module load&amp;quot; command for the engineering applications we provide&lt;br /&gt;
# on ARROW will handle all those details transparently without the user needing to understand the details.&lt;br /&gt;
export I_MPI_HYDRA_BOOTSTRAP=ssh&lt;br /&gt;
export MPI_DEVICE=rdma:ofa-v2-ib0&lt;br /&gt;
export UCX_NET_DEVICES=qib0:1&lt;br /&gt;
#&lt;br /&gt;
# it doesn&#039;t make much sense, but in this example we are executing the OS command &amp;quot;uptime&amp;quot; on all cores&lt;br /&gt;
# of the nodes allocated to this job. The output from each core is written to the file info.log. We&lt;br /&gt;
# will find 56 lines of output in the file info.log, each created by the corresponding core executing&lt;br /&gt;
# the uptime command.&lt;br /&gt;
mpirun uptime &amp;gt; info.log&lt;br /&gt;
#&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
If the resource requests are embedded within the file, they don&#039;t have to be specified on the command line any longer (the command line overrides the embedded specifications though). This may be convenient, because all the user has to do for job submission is the following:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
qsub parallel.job&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
Here is an example with more resource specifications and job settings that affect the behavior of the job:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
#!/bin/bash&lt;br /&gt;
#&lt;br /&gt;
#PBS -q xeon28&lt;br /&gt;
#PBS -l walltime=1:0:0&lt;br /&gt;
#PBS -l select=2:ncpus=28:mpiprocs=28:mem=60G&lt;br /&gt;
#PBS -A Account&lt;br /&gt;
#PBS -j oe&lt;br /&gt;
#PBS -N JobName&lt;br /&gt;
#PBS -e log.error&lt;br /&gt;
#PBS -o log.output&lt;br /&gt;
#PBS -m bae&lt;br /&gt;
#&lt;br /&gt;
...&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
I leave this to you as an exercise to figure out what the various options mean and how to specify them. There are many more, all documented in the PBS PRO manual (see above). Most of them are not terribly relevant and can be omitted.&lt;br /&gt;
&lt;br /&gt;
===Interactive Jobs===&lt;br /&gt;
&lt;br /&gt;
On ARROW, we don&#039;t restrict queues to be used only in batch mode. While &#039;&#039;&#039;batch mode&#039;&#039;&#039; is efficient for lining up a lot of work to be executed one after the other, ARROW has been designed to &#039;&#039;&#039;allow efficient model and software development in interactive mode&#039;&#039;&#039;. We have always ensured to have more computers than minimally needed to make it possible to dedicate resources to developers as needed, even if that means wasted CPU cycles. At times, we may ask you to limit the number of interactive jobs so that a large batch workload can be processed efficiently. This happens from time to time, and we have our users coordinate this with each other.&lt;br /&gt;
&lt;br /&gt;
Let&#039;s assume that you are developing an MPI application, or you are working on a complex &#039;&#039;&#039;OpenFOAM&#039;&#039;&#039; model that requires to start parallel processes over and over again just to find a bug and then fix it quickly. To do that, you can &#039;&#039;&#039;request an interactive job&#039;&#039;&#039; by adding the &#039;&#039;&#039;&amp;quot;-I&amp;quot;&#039;&#039;&#039; option to the job submission command (this is an uppercase I). Let&#039;s go to the parallel multi-node example from above:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
qsub -I -q xeon28 -l walltime=1:0:0 -l select=2:ncpus=28:mpiprocs=28:mem=60G&lt;br /&gt;
      ^    ^                  ^               ^       ^           ^      ^ ^&lt;br /&gt;
      |    |                  |               |       |           |      | + --- don&#039;t forget to specify gigabytes&lt;br /&gt;
      |    |                  |               |       |           |      + ----- the amount of memory to request per node&lt;br /&gt;
      |    |                  |               |       |           + ------------ the number of MPI tasks per nodes&lt;br /&gt;
      |    |                  |               |       + ------------------------ the number of cores per node&lt;br /&gt;
      |    |                  |               + -------------------------------- the number of nodes to select in the queue&lt;br /&gt;
      |    |                   + ----------------------------------------------- the requested time, in this case 1h&lt;br /&gt;
      |    + ------------------------------------------------------------------- the queue to be used for the job&lt;br /&gt;
      + ------------------------------------------------------------------------ request an interactive job &amp;lt;&amp;lt;===&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
When running interactive jobs with the &#039;&#039;&#039;&amp;quot;-I&amp;quot;&#039;&#039;&#039; parameter, we don&#039;t specify av job script at the end of the submission command. The interactive job will instead start (once the nodes are available) in interactive mode, meaning that the terminal session changes over from being a series of commands executed on the login server to being a series of commands being executed on the first node of the group of nodes that are allocated to the job. At this point, you can change to the desired working directory, but what you do with the allocated resources is entirely up to you. You can load modules, including MPI libraries, and then issue the commands for your application interactively and see how they execute. If you start an &#039;&#039;&#039;&amp;quot;mpirun&amp;quot;&#039;&#039;&#039;, the cores on your allocated secondary node will work as expected. There is no difference to batch mode, other than you having the ability to execute lines of commands at will.&lt;br /&gt;
&lt;br /&gt;
===Interactive Jobs with X-Windows GUI Applications===&lt;br /&gt;
&lt;br /&gt;
Interactive use can go further than that. With some of our software applications, like &#039;&#039;&#039;StarCCM+&#039;&#039;&#039;, you can run an &#039;&#039;&#039;interactive GUI application&#039;&#039;&#039; where you control the computational work from within the applications&#039; GUI. Within the GUI, you can control execution of the numerical solver that runs on as many cores as you requested, while being able to reconfigure the case through the GUI as well. Furthermore, you can visualize developing results on the fly by creating complex plots and visualizations.&lt;br /&gt;
&lt;br /&gt;
All that is need is an option &#039;&#039;&#039;&amp;quot;-X&amp;quot;&#039;&#039;&#039; being used as part of the job submission, like this:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
qsub -X -I -q xeon28 -l walltime=1:0:0 -l select=2:ncpus=28:mpiprocs=28:mem=60G&lt;br /&gt;
      ^  ^    ^                  ^               ^       ^           ^      ^ ^&lt;br /&gt;
      |  |    |                  |               |       |           |      | + --- don&#039;t forget to specify gigabytes&lt;br /&gt;
      |  |    |                  |               |       |           |      + ----- the amount of memory to request per node&lt;br /&gt;
      |  |    |                  |               |       |           + ------------ the number of MPI tasks per nodes&lt;br /&gt;
      |  |    |                  |               |       + ------------------------ the number of cores per node&lt;br /&gt;
      |  |    |                  |               + -------------------------------- the number of nodes to select in the queue&lt;br /&gt;
      |  |    |                   + ----------------------------------------------- the requested time, in this case 1h&lt;br /&gt;
      |  |    + ------------------------------------------------------------------- the queue to be used for the job&lt;br /&gt;
      |  + ------------------------------------------------------------------------ request an interactive job&lt;br /&gt;
      + --------------------------------------------------------------------------- request GUI capabilities &amp;lt;&amp;lt;===&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
===Running Multiple Jobs on Single Nodes===&lt;br /&gt;
&lt;br /&gt;
A feature that is new on ARROW is the ability to run multiple jobs on a single node. Let&#039;s assume that you are performing a sensitivity analysis on an existing model, and the model is simple enough to return results within a reasonable time on just a few cores of a higher end machine (maybe you are running SMP versions of &#039;&#039;&#039;LS-Dyna&#039;&#039;&#039;). Our high end machines have 64 cores, so lets assume you have an &#039;&#039;&#039;LS-Dyna&#039;&#039;&#039; model that runs well on 8 cores and doesn&#039;t use a whole lot of memory. In this case, you can submit individual jobs that request simply 8 cores and a fraction of the available memory available on the node, and all jobs execute independently from each other. Each job is fit into a slot where available. It is not very different from using whole nodes for everything. The important consideration is that each job is cleanly constrained into it&#039;s allotted resources using the &#039;&#039;&#039;CGROUPS&#039;&#039;&#039; functionality of modern operating systems. Because an abusive user cannot use more cores or more memory than allocated to his job, other users can safely run smaller jobs on the same node.&lt;br /&gt;
&lt;br /&gt;
Lets assume that we have a number of smaller jobs that we want to run on a single node in the &#039;&#039;&#039;&amp;quot;xeon28&amp;quot;&#039;&#039;&#039; queue. Each job would be submitted by using reduced resources that allow for sharing but that  guarantee that the jobs will be run successfully. In this case, you can &#039;&#039;&#039;submit many jobs&#039;&#039;&#039; in the following manner (with a job script for the small jobs, each of which can &#039;&#039;&#039;request varying resources&#039;&#039;&#039; if needed - some may want to run on 5 cores, others on 3):&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
#!/bin/bash&lt;br /&gt;
#&lt;br /&gt;
# the following ensures that you will change into the directory where you are&lt;br /&gt;
# submitting the job from.&lt;br /&gt;
cd $PBS_O_WORKDIR&lt;br /&gt;
#&lt;br /&gt;
# now we sleep for 300 seconds and waste time. This is a placeholder for your application,&lt;br /&gt;
# which would be doing useful work for you.&lt;br /&gt;
sleep 300&lt;br /&gt;
#&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
Now we submit a variety of these jobs (11 total in this example) to the &#039;&#039;&#039;&amp;quot;xeon28&amp;quot;&#039;&#039;&#039; queue for execution (note that the first few jobs request different amounts of memory and core counts):&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
qsub -q xeon28 -l walltime=1:0:0 -l select=1:ncpus=12:mpiprocs=12:mem=5G small.job &lt;br /&gt;
qsub -q xeon28 -l walltime=1:0:0 -l select=1:ncpus=10:mpiprocs=10:mem=7G small.job &lt;br /&gt;
qsub -q xeon28 -l walltime=1:0:0 -l select=1:ncpus=8:mpiprocs=8:mem=9G small.job &lt;br /&gt;
qsub -q xeon28 -l walltime=1:0:0 -l select=1:ncpus=16:mpiprocs=16:mem=20G small.job &lt;br /&gt;
qsub -q xeon28 -l walltime=1:0:0 -l select=1:ncpus=2:mpiprocs=2:mem=2G small.job &lt;br /&gt;
qsub -q xeon28 -l walltime=1:0:0 -l select=1:ncpus=2:mpiprocs=2:mem=2G small.job &lt;br /&gt;
qsub -q xeon28 -l walltime=1:0:0 -l select=1:ncpus=2:mpiprocs=2:mem=2G small.job &lt;br /&gt;
qsub -q xeon28 -l walltime=1:0:0 -l select=1:ncpus=2:mpiprocs=2:mem=2G small.job &lt;br /&gt;
qsub -q xeon28 -l walltime=1:0:0 -l select=1:ncpus=2:mpiprocs=2:mem=2G small.job &lt;br /&gt;
qsub -q xeon28 -l walltime=1:0:0 -l select=1:ncpus=2:mpiprocs=2:mem=2G small.job &lt;br /&gt;
qsub -q xeon28 -l walltime=1:0:0 -l select=1:ncpus=2:mpiprocs=2:mem=2G small.job &lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
They are now running in the order of submission, allocated on as few nodes in the &amp;quot;xeon28&amp;quot; queue as necessary. Only 2 nodes are being loaded quite heavily, and 4 more cores are in use on a third node.&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
Nov 20 23:34 ley@login3:myjobfolder$ qstat -an1&lt;br /&gt;
&lt;br /&gt;
pbs: &lt;br /&gt;
                                                            Req&#039;d  Req&#039;d   Elap&lt;br /&gt;
Job ID          Username Queue    Jobname    SessID NDS TSK Memory Time  S Time&lt;br /&gt;
--------------- -------- -------- ---------- ------ --- --- ------ ----- - -----&lt;br /&gt;
&lt;br /&gt;
3082.pbs        ley      xeon28   small.job  813221   1  12    5gb 01:00 R 00:01 p001/0*12&lt;br /&gt;
3083.pbs        ley      xeon28   small.job  813288   1  10    7gb 01:00 R 00:01 p001/1*10&lt;br /&gt;
3084.pbs        ley      xeon28   small.job  671792   1   8    9gb 01:00 R 00:01 p002/0*8&lt;br /&gt;
3085.pbs        ley      xeon28   small.job  671845   1  16   20gb 01:00 R 00:01 p002/1*16&lt;br /&gt;
3086.pbs        ley      xeon28   small.job  813361   1   2    2gb 01:00 R 00:00 p001/2*2&lt;br /&gt;
3087.pbs        ley      xeon28   small.job  813413   1   2    2gb 01:00 R 00:00 p001/3*2&lt;br /&gt;
3088.pbs        ley      xeon28   small.job  813464   1   2    2gb 01:00 R 00:00 p001/4*2&lt;br /&gt;
3089.pbs        ley      xeon28   small.job  671912   1   2    2gb 01:00 R 00:00 p002/2*2&lt;br /&gt;
3090.pbs        ley      xeon28   small.job  671969   1   2    2gb 01:00 R 00:00 p002/3*2&lt;br /&gt;
3091.pbs        ley      xeon28   small.job  632092   1   2    2gb 01:00 R 00:00 p003/0*2&lt;br /&gt;
3092.pbs        ley      xeon28   small.job  632100   1   2    2gb 01:00 R 00:00 p003/1*2&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
This is a particularly effective strategy to run concurrently many cases that don&#039;t scale well beyond a few cores. When running them on fewer cores but many of them at the same time, the overall processing rate will be much higher than executing them the traditional way.&lt;br /&gt;
&lt;br /&gt;
===Running Jobs using GPUs===&lt;br /&gt;
&lt;br /&gt;
The principle of running multiple jobs on a single node becomes particularly important when using servers equipped with &#039;&#039;&#039;GPUs&#039;&#039;&#039; for &#039;&#039;&#039;ML/AI&#039;&#039;&#039; applications. The cluster doesn&#039;t have a whole lot of &#039;&#039;&#039;GPUs&#039;&#039;&#039; at this point. We have three machines with three &#039;&#039;&#039;A4000&#039;&#039;&#039; GOUs, a &#039;&#039;&#039;total of 9 A4000 GPUs&#039;&#039;&#039;. Then we have a much more powerful single machine with our &#039;&#039;&#039;four A6000 GPUs&#039;&#039;&#039;.&lt;br /&gt;
&lt;br /&gt;
Using multiple GPUs in a single application is still something where the software has to be designed with hardware in mind. GPUs have several methods of communicating with each other, e.g. very fast &#039;&#039;&#039;NVLINK&#039;&#039;&#039; between pairs of GPUs, GPUs being directly connected to one of the two CPUs in the system and thus being able to communicate faster, and GPUs that have to jump between processors when communicating, and then the whole issue of having to go possibly through PCIe bridges.&lt;br /&gt;
&lt;br /&gt;
On our system, we are providing the ability to &#039;&#039;&#039;work mostly with individual GPUs&#039;&#039;&#039;. Users can also reserve entire nodes and develop or run applications that are adapted to that hardware, including several GPUs installed on that node. One thing we do not provide is the ability of GPU to GPU communication between nodes. Thus, a job cannot request more than one GPU node at a time.&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
qsub -q a4000 -I -l walltime=1:0:0 -l select=1:ncpus=5:mem=150G:ngpus=1&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
With these specifications, three single GPU jobs can run on a single server. Each job sees only one of the reserved GPU.&lt;br /&gt;
&lt;br /&gt;
To run a massive GPU job on 64 cores with 4 &#039;&#039;&#039;A6000 GPUs&#039;&#039;&#039;, submit the job like this:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
qsub -q a6000 -I -l walltime=1:0:0 -l select=1:ncpus=64:mem=725G:ngpus=4&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
===Manual pages for qsub===&lt;br /&gt;
&lt;br /&gt;
To learn more about the &amp;quot;qsub&amp;quot; command, you can use the command &amp;quot;man qsub&amp;quot;, which will print a lot of detailed information about the capabilities of this command.&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
$ man qsub&lt;br /&gt;
&lt;br /&gt;
qsub(1B)                                               PBS Professional                                              qsub(1B)&lt;br /&gt;
&lt;br /&gt;
NAME&lt;br /&gt;
       qsub - submit a job to PBS&lt;br /&gt;
&lt;br /&gt;
SYNOPSIS&lt;br /&gt;
       qsub [-a &amp;lt;date and time&amp;gt;] [-A &amp;lt;account string&amp;gt;] [-c &amp;lt;checkpoint spec&amp;gt;]&lt;br /&gt;
            [-C &amp;lt;directive prefix&amp;gt;] [-e &amp;lt;path&amp;gt;] [-f] [-h]&lt;br /&gt;
            [-I [-G [-- &amp;lt;GUI application/script&amp;gt;]] | [-X]] [-j &amp;lt;join&amp;gt;]&lt;br /&gt;
            [-J &amp;lt;range&amp;gt; [%&amp;lt;max subjobs]] [-k &amp;lt;discard&amp;gt;] [-l &amp;lt;resource list&amp;gt;]&lt;br /&gt;
            [-m &amp;lt;mail events&amp;gt;] [-M &amp;lt;user list&amp;gt;] [-N &amp;lt;name&amp;gt;] [-o &amp;lt;path&amp;gt;]&lt;br /&gt;
            [-p &amp;lt;priority&amp;gt;] [-P &amp;lt;project&amp;gt;] [-q &amp;lt;destination&amp;gt;] [-r &amp;lt;y|n&amp;gt;]&lt;br /&gt;
            [-R &amp;lt;remove options&amp;gt;] [-S &amp;lt;path list&amp;gt;] [-u &amp;lt;user list&amp;gt;]&lt;br /&gt;
            [-v &amp;lt;variable list&amp;gt;] [-V] [-W &amp;lt;additional attributes&amp;gt;] [-z]&lt;br /&gt;
            [- | &amp;lt;script&amp;gt; | -- &amp;lt;executable&amp;gt; [&amp;lt;arguments to executable&amp;gt;]]&lt;br /&gt;
       qsub --version&lt;br /&gt;
&lt;br /&gt;
DESCRIPTION&lt;br /&gt;
       You use the qsub command to submit a batch job to PBS.  Submitting a PBS job specifies a task, requests resources, and&lt;br /&gt;
       sets job attributes.&lt;br /&gt;
... &amp;lt;many more pages&amp;gt;&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
==LS-Dyna on the ARROW Cluster==&lt;br /&gt;
&lt;br /&gt;
===Currently Available LS-Dyna Versions===&lt;br /&gt;
&lt;br /&gt;
The following is a list of &#039;&#039;&#039;LS-Dyna versions&#039;&#039;&#039; available on &#039;&#039;&#039;ARROW&#039;&#039;&#039; after the latest reconfiguration of the system. As per LSTC/ANSYS, &#039;&#039;&#039;versions before 14.0.0 are not necessarily fully supported any longer&#039;&#039;&#039; because they are supposedly not compatible with modern operating systems and cannot be made to work reliably. We have tested the listed older versions of LS-Dyna and they have passed basic tests. They may not behave exactly as they did on the old CentOS 7 operating system, and time will show whether they can still be used or whether you will need to update your models and use a fully supported version.&lt;br /&gt;
&lt;br /&gt;
All versions are loaded using the &#039;&#039;&#039;&amp;quot;module load&amp;quot;&#039;&#039;&#039; command. Versions can be listed with the &#039;&#039;&#039;&amp;quot;module avail ls-dyna&amp;quot;&#039;&#039;&#039; command. To load one of the modules, use the following syntax:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
module load ls-dyna/14.2.0/mpi-d8-ifort190-avx512&lt;br /&gt;
            ^       ^      ^   ^  ^        ^&lt;br /&gt;
            |       |      |   |  |        + --- specify the extended instruction set needed for execution&lt;br /&gt;
            |       |      |   |  + ------------ load the version of the compiler that was used to create this &lt;br /&gt;
            |       |      |   + --------------- load the version that supports double precision variables&lt;br /&gt;
            |       |      + ------------------- load the MPP (MPI) version of LS-Dyna&lt;br /&gt;
            |       + -------------------------- load specifically version 14.2.0&lt;br /&gt;
            + ---------------------------------- load a version of LS-Dyna&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
The version string is composed of multiple elements to indicate variants in compilers and compiler options. Use the following guideline to choose an appropriate version to load:&lt;br /&gt;
&lt;br /&gt;
* &#039;&#039;&#039;&amp;quot;1&amp;quot;&#039;&#039;&#039; or &#039;&#039;&#039;&amp;quot;mpi&amp;quot;&#039;&#039;&#039; indicates whether this is a single node version of LS-Dyna (&#039;&#039;&#039;SMP&#039;&#039;&#039;) or whether this is a multi-node MPI version (&#039;&#039;&#039;MPP&#039;&#039;&#039;). All MPI versions use the &#039;&#039;&#039;IntelMPI 2022&#039;&#039;&#039; libraries which have been tested thoroughly on ARROW. MPI versions will use the Infiniband Network of ARROW for high-speed and low-latency inter-process communication using RDMA (remote direct memory access).&lt;br /&gt;
* All LS-Dyna versions are available in either &#039;&#039;&#039;floating point&#039;&#039;&#039; or &#039;&#039;&#039;double precision variants&#039;&#039;&#039;. Floating point variants use 4 bytes to represent a value, and double precision variants use 8 bytes. There are pros and cons for choosing one over the other variant. With regards to computational efficiency, both perform nearly the same because all machines are equipped with 64-bit CPUs.&lt;br /&gt;
** &#039;&#039;&#039;&amp;quot;f4&amp;quot;&#039;&#039;&#039; floating point versions&lt;br /&gt;
*** Pros: These require significantly less memory to run. Results occupy less disk space, and can be transferred significantly faster into and out of ARROW.&lt;br /&gt;
*** Cons: The numerical resolution is limited to 7 significant digits, which is often undesirable when dealing with mathematical operations on small and large numbers at the same time.&lt;br /&gt;
** &#039;&#039;&#039;&amp;quot;r8&amp;quot;&#039;&#039;&#039; double precision versions&lt;br /&gt;
*** Pros: The numerical resolution is about twice the number of significant digits compare to &amp;quot;f4&amp;quot;, which helps when when dealing with mathematical operations on small and large numbers at the same time.&lt;br /&gt;
*** Cons: These require more memory to run. Results occupy more disk space, and it takes longer to transfer data into and out of ARROW.&lt;br /&gt;
* There are two more identifiers to choose from when it comes to the variants of the executables: the specific compiler used to create the executable and the specific processor instruction set required for running the executable.&lt;br /&gt;
** For modern versions of LS-Dyna, two compilers have been used by the developers to create LS-Dyna executables: the &#039;&#039;&#039;Intel Fortran Compiler&#039;&#039;&#039; and the &#039;&#039;&#039;AOCC (AMD Optimizing C/C++ and Fortran) compiler&#039;&#039;&#039;. Both variants of the software are supported on ARROW. This gives users the opportunity to choose an alternate variant of the same LS-Dyna version when running into bugs or crashes.&lt;br /&gt;
** The variants based on the various instruction set extensions (&#039;&#039;&#039;SSE2&#039;&#039;&#039;, &#039;&#039;&#039;AVX2&#039;&#039;&#039;, &#039;&#039;&#039;AVX512&#039;&#039;&#039;, and so on) gives users even more options when choosing an alternate LS-Dyna variant of the same version when running into bugs or crashes. These instruction sets are mostly related to performance gains on specific processors. We have not performed thorough performance tests and cannot recommend specific versions right now.&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
$ module avail ls-dyna&lt;br /&gt;
--------------------------------------------- /shared/apps/modulefiles ---------------------------------------------&lt;br /&gt;
ls-dyna/09.3.1/1-d8-ifort131           ls-dyna/12.2.1/mpi-f4-ifort160-sse2    ls-dyna/14.2.0/mpi-f4-ifort190-avx512  &lt;br /&gt;
ls-dyna/09.3.1/1-f4-ifort131           ls-dyna/12.2.2/1-d8-ifort160-sse2      ls-dyna/14.2.0/mpi-f4-ifort190-sse2    &lt;br /&gt;
ls-dyna/09.3.1/mpi-d8-ifort131-avx2    ls-dyna/12.2.2/1-f4-ifort160-sse2      ls-dyna/15.0.2/1-d8-ifort190-sse2      &lt;br /&gt;
ls-dyna/09.3.1/mpi-d8-ifort131-avx512  ls-dyna/12.2.2/mpi-d8-aocc400-avx2     ls-dyna/15.0.2/1-f4-ifort190-sse2      &lt;br /&gt;
ls-dyna/09.3.1/mpi-f4-ifort131-avx2    ls-dyna/12.2.2/mpi-d8-ifort160-avx2    ls-dyna/15.0.2/mpi-d8-aocc400-avx2     &lt;br /&gt;
ls-dyna/09.3.1/mpi-f4-ifort131-avx512  ls-dyna/12.2.2/mpi-d8-ifort160-sse2    ls-dyna/15.0.2/mpi-d8-ifort190-avx2    &lt;br /&gt;
ls-dyna/10.2.0/1-d8-ifort160           ls-dyna/12.2.2/mpi-f4-aocc400-avx2     ls-dyna/15.0.2/mpi-d8-ifort190-avx512  &lt;br /&gt;
ls-dyna/10.2.0/1-f4-ifort160           ls-dyna/12.2.2/mpi-f4-ifort160-avx2    ls-dyna/15.0.2/mpi-d8-ifort190-sse2    &lt;br /&gt;
ls-dyna/11.0.0/1-d8-ifort160           ls-dyna/12.2.2/mpi-f4-ifort160-sse2    ls-dyna/15.0.2/mpi-f4-aocc400-avx2     &lt;br /&gt;
ls-dyna/11.0.0/1-f4-ifort160           ls-dyna/13.0.0/1-d8-ifort190           ls-dyna/15.0.2/mpi-f4-ifort190-avx2    &lt;br /&gt;
ls-dyna/11.1.0/1-d8-ifort160-sse2      ls-dyna/13.0.0/1-f4-ifort190           ls-dyna/15.0.2/mpi-f4-ifort190-avx512  &lt;br /&gt;
ls-dyna/11.1.0/1-f4-ifort160-sse2      ls-dyna/13.0.0/mpi-d8-ifort190-avx2    ls-dyna/15.0.2/mpi-f4-ifort190-sse2    &lt;br /&gt;
ls-dyna/11.2.0/1-d8-ifort160           ls-dyna/13.0.0/mpi-d8-ifort190-sse2    ls-dyna/16.0.0/1-d8-aocc420-avx2       &lt;br /&gt;
ls-dyna/11.2.0/1-f4-ifort160           ls-dyna/13.0.0/mpi-f4-ifort190-avx2    ls-dyna/16.0.0/1-d8-aocc420-avx512     &lt;br /&gt;
ls-dyna/11.2.0/mpi-f4-ifort160-avx2    ls-dyna/13.0.0/mpi-f4-ifort190-sse2    ls-dyna/16.0.0/1-d8-ifort190-sse2      &lt;br /&gt;
ls-dyna/11.2.0/mpi-f4-ifort160-sse2    ls-dyna/13.1.0/mpi-d8-aocc310-avx2     ls-dyna/16.0.0/1-f4-aocc420-avx2       &lt;br /&gt;
ls-dyna/11.2.1/1-d8-ifort160           ls-dyna/13.1.0/mpi-d8-ifort190-avx2    ls-dyna/16.0.0/1-f4-aocc420-avx512     &lt;br /&gt;
ls-dyna/11.2.1/1-f4-ifort160           ls-dyna/13.1.0/mpi-d8-ifort190-sse2    ls-dyna/16.0.0/1-f4-ifort190-sse2      &lt;br /&gt;
ls-dyna/11.2.1/mpi-d8-ifort160-avx2    ls-dyna/13.1.0/mpi-f4-aocc310-avx2     ls-dyna/16.0.0/mpi-d8-aocc420-avx2     &lt;br /&gt;
ls-dyna/11.2.1/mpi-d8-ifort160-sse2    ls-dyna/13.1.0/mpi-f4-ifort190-avx2    ls-dyna/16.0.0/mpi-d8-aocc420-avx512   &lt;br /&gt;
ls-dyna/11.2.1/mpi-f4-ifort160-avx2    ls-dyna/13.1.0/mpi-f4-ifort190-sse2    ls-dyna/16.0.0/mpi-d8-ifort190-avx2    &lt;br /&gt;
ls-dyna/11.2.1/mpi-f4-ifort160-sse2    ls-dyna/13.1.1/mpi-d8-ifort190-avx2    ls-dyna/16.0.0/mpi-d8-ifort190-avx512  &lt;br /&gt;
ls-dyna/11.2.2/mpi-d8-ifort160-avx2    ls-dyna/13.1.1/mpi-d8-ifort190-sse2    ls-dyna/16.0.0/mpi-d8-ifort190-sse2    &lt;br /&gt;
ls-dyna/11.2.2/mpi-d8-ifort160-sse2    ls-dyna/13.1.1/mpi-f4-ifort190-avx2    ls-dyna/16.0.0/mpi-f4-aocc420-avx2     &lt;br /&gt;
ls-dyna/11.2.2/mpi-f4-ifort160-avx2    ls-dyna/13.1.1/mpi-f4-ifort190-sse2    ls-dyna/16.0.0/mpi-f4-aocc420-avx512   &lt;br /&gt;
ls-dyna/11.2.2/mpi-f4-ifort160-sse2    ls-dyna/14.0.0/1-d8-ifort190           ls-dyna/16.0.0/mpi-f4-ifort190-avx2    &lt;br /&gt;
ls-dyna/12.1.0/1-d8-ifort160           ls-dyna/14.0.0/1-f4-ifort190           ls-dyna/16.0.0/mpi-f4-ifort190-avx512  &lt;br /&gt;
ls-dyna/12.1.0/1-f4-aocc310            ls-dyna/14.0.0/mpi-d8-aocc310-avx2     ls-dyna/16.0.0/mpi-f4-ifort190-sse2    &lt;br /&gt;
ls-dyna/12.1.0/1-f4-ifort160           ls-dyna/14.0.0/mpi-d8-ifort190-avx2    ls-dyna/16.1.0/mpi-d8-aocc420-avx2     &lt;br /&gt;
ls-dyna/12.1.0/mpi-d8-aocc310-avx2     ls-dyna/14.0.0/mpi-d8-ifort190-sse2    ls-dyna/16.1.0/mpi-d8-aocc420-avx512   &lt;br /&gt;
ls-dyna/12.1.0/mpi-d8-ifort160-avx2    ls-dyna/14.0.0/mpi-f4-ifort190-avx2    ls-dyna/16.1.0/mpi-d8-ifort190-avx2    &lt;br /&gt;
ls-dyna/12.1.0/mpi-d8-ifort160-sse2    ls-dyna/14.0.0/mpi-f4-ifort190-sse2    ls-dyna/16.1.0/mpi-d8-ifort190-avx512  &lt;br /&gt;
ls-dyna/12.1.0/mpi-f4-aocc310-avx2     ls-dyna/14.1.0/1-d8-ifort190-sse2      ls-dyna/16.1.0/mpi-d8-ifort190-sse2    &lt;br /&gt;
ls-dyna/12.1.0/mpi-f4-ifort160-avx2    ls-dyna/14.1.0/1-f4-ifort190-sse2      ls-dyna/16.1.0/mpi-f4-aocc420-avx2     &lt;br /&gt;
ls-dyna/12.1.0/mpi-f4-ifort160-sse2    ls-dyna/14.1.0/mpi-d8-aocc400-avx2     ls-dyna/16.1.0/mpi-f4-aocc420-avx512&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
===Submitting an LS-Dyna Job===&lt;br /&gt;
&lt;br /&gt;
&amp;lt;BLOCKQUOTE&amp;gt;&lt;br /&gt;
[[File:Attention.jpg|25px]] &#039;&#039;&#039;IMPORTANT NOTE:&#039;&#039;&#039; The job/queue manager can now track the number of LS-Dyna licenses given out to individual&lt;br /&gt;
jobs. At submission time, it is not possible to know what software a user may run. But by adding the clause &amp;quot;-l dynalic&amp;quot; at submission time,&lt;br /&gt;
the queue manager can calculate the total number of cores required and keep track of LS-Dyna licenses used by the job. When loading a version of LS-Dyna, a check will be performed, and LS-Dyna will be prevented from running if the &amp;quot;-l dynalic&amp;quot; clause was not used when submitting the job.&#039;&#039;&lt;br /&gt;
&lt;br /&gt;
&amp;lt;!--&lt;br /&gt;
&#039;&#039;The job/queue manager can track the number of LS-Dyna licenses to some degree. If all &#039;&#039;&#039;LS-Dyna users&#039;&#039;&#039; cooperate and use a script like the one shown below when submitting their jobs, the total number of concurrent &#039;&#039;&#039;LS-Dyna licenses&#039;&#039;&#039; will be tracked by the job manager correctly. That means that users can submit any number of LS-Dyna jobs, and jobs will only start when a sufficient number of licenses is available. This is managed by the &#039;&#039;&#039;&amp;quot;dynalic&amp;quot;&#039;&#039;&#039; resource at the end of the select statement. In this example, a 2-node job on 64-core nodes will need a total of &#039;&#039;&#039;&amp;quot;dynalic=128&amp;quot;&#039;&#039;&#039; licenses. This accounting breaks down when users don&#039;t use the &#039;&#039;&#039;&amp;quot;dynalic=XXX&amp;quot;&#039;&#039;&#039; statement, or when they don&#039;t calculate the number of licenses correctly. In that case, LS-Dyna jobs of all users are subject to sudden failure when LS-Dyna licenses run out. Please understand the importance of this specific setting in your job.&#039;&#039;&lt;br /&gt;
--&amp;gt;&lt;br /&gt;
&amp;lt;/BLOCKQUOTE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
Furthermore, careful consideration should be given with regards to choice of resources for an &#039;&#039;&#039;LS-Dyna job&#039;&#039;&#039;. With 64 cores available on a single node in the &#039;&#039;&#039;&amp;quot;epyc1&amp;quot;&#039;&#039;&#039; and &#039;&#039;&#039;&amp;quot;epyc2&amp;quot;&#039;&#039;&#039; queues, it may be counterproductive to run a job on two nodes instead of a single node. Users should run their jobs with different numbers of nodes and determine whether performance increases. It may well decrease when running a job on two or more nodes. The outcome of such tests will tell what the best allocation of resources will be.&lt;br /&gt;
&lt;br /&gt;
Most users use a job script like the following. All methods for job submission the the previous chapters apply as well, so there is a lot of flexibility:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
#!/bin/bash&lt;br /&gt;
#&lt;br /&gt;
#PBS -q epyc1&lt;br /&gt;
#PBS -l walltime=12:0:0&lt;br /&gt;
#PBS -l select=2:ncpus=64:mpiprocs=64:mem=225G,dynalic=128&lt;br /&gt;
#PBS -N JobName&lt;br /&gt;
#PBS -e log.error&lt;br /&gt;
#PBS -o log.output&lt;br /&gt;
#&lt;br /&gt;
cd $PBS_O_WORKDIR&lt;br /&gt;
#&lt;br /&gt;
module load ls-dyna/12.2.1/mpi-f4-ifort160-avx2&lt;br /&gt;
module load dynamore/current&lt;br /&gt;
#&lt;br /&gt;
mpirun ls-dyna i=main.k memory1=300m memory2=100m &amp;gt; dyna.log&lt;br /&gt;
#&lt;br /&gt;
# when using the Dynamore tools, you can start something like this at the end&lt;br /&gt;
DM.plotcprs.lnx -merge &amp;gt;&amp;gt; dyna.log&lt;br /&gt;
#&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
===LSTC Tools: LS-OPT and LS-PREPOST===&lt;br /&gt;
&lt;br /&gt;
For the new Rocky 9 cluster, I have not looked deeply into the ls-opt and ls-prepost packages that were installed. I noticed though that the LSTC server provided access to much newer versions of both software packages. If you would like to learn more or have a specific version in mind, I can easily download and install it for you.&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
$ module avail ls-opt&lt;br /&gt;
----------------------------------------------- /shared/apps/modulefiles ------------------------------------------------&lt;br /&gt;
ls-opt/5.1.1  ls-opt/6.0.0  ls-opt/7.0.0  ls-opt/7.0.2   ls-opt/2022R2  &lt;br /&gt;
ls-opt/5.2.1  ls-opt/6.1.0  ls-opt/7.0.1  ls-opt/2022R1  ls-opt/2023R1  &lt;br /&gt;
&lt;br /&gt;
$ module avail ls-prepost&lt;br /&gt;
----------------------------------------------- /shared/apps/modulefiles ------------------------------------------------&lt;br /&gt;
ls-prepost/4.5.10  ls-prepost/4.8.13  ls-prepost/4.8.30  ls-prepost/4.9.16  ls-prepost/4.10.7 &lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
===Dynamore Software===&lt;br /&gt;
&lt;br /&gt;
The Dynamore tools are available as a module:&lt;br /&gt;
&lt;br /&gt;
 module load dynamore/current&lt;br /&gt;
&lt;br /&gt;
We typically acquire a yearly license for the tools as we purchase licenses for LS-Dyna.&lt;br /&gt;
&lt;br /&gt;
===Vendor License File Installation===&lt;br /&gt;
&lt;br /&gt;
If you would like for us to install a vendor license for LS-Dyna models, please contact us for the required information. We can send you the general LS-Dyna license file to show the host ids for the license server. Using that information, your vendor should be able to create a vendor license file. Please send that file to us per Email or by other means.&lt;br /&gt;
&lt;br /&gt;
==StarCCM+ on the ARROW Cluster==&lt;br /&gt;
&lt;br /&gt;
===Currently Available StarCCM+ Versions===&lt;br /&gt;
&lt;br /&gt;
As of late 2025, we have the following versions of &#039;&#039;&#039;StarCCM+&#039;&#039;&#039; available on the cluster:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
$ module avail starccm&lt;br /&gt;
---------------------------- /shared/apps/modulefiles ----------------------------&lt;br /&gt;
starccm/15.02.007-R8  starccm/16.06.008-R8  starccm/18.06.006-R8  &lt;br /&gt;
starccm/15.02.009-R8  starccm/17.02.007-R8  starccm/19.02.009-R8  &lt;br /&gt;
starccm/15.04.008-R8  starccm/17.02.008-R8  starccm/20.04.007-R8  &lt;br /&gt;
starccm/15.06.008-R8  starccm/17.04.007-R8  starccm/20.06.007-R8  &lt;br /&gt;
starccm/16.02.008-R8  starccm/17.06.007-R8  &lt;br /&gt;
starccm/16.04.007-R8  starccm/18.04.008-R8&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
If using a &#039;&#039;&#039;single node&#039;&#039;&#039; for StarCCM+, job submission (for an interactive job) is simple and will use appropriate default settings:&lt;br /&gt;
&lt;br /&gt;
 qsub -I -X -q epyc1 -l walltime=20:00:00&lt;br /&gt;
&lt;br /&gt;
StarCCM+ can make use of the job scheduler attributes by automatically obtaining the number of cores and other resources from OpenPBS. In this case, the default number of cores and mpi processes for StarCCM+ are both 64 when using the epyc1 queue. So you can start your StarCCM+ run with:&lt;br /&gt;
&lt;br /&gt;
 module load starccm/15.02.007-R8 (or any other version)&lt;br /&gt;
 starccm+ -bs pbs&lt;br /&gt;
&lt;br /&gt;
In this case, there is no need for StarCCM+ to be told to run the case in parallel with the selected number of cores/mpiprocs.&lt;br /&gt;
&lt;br /&gt;
This can get a bit more complex when running on multiple nodes or when requesting high memory nodes. In that case you would use job submission parameters as shown below:&lt;br /&gt;
&lt;br /&gt;
 qsub -I -X -q epyc1 -l walltime=20:00:00,select=2:ncpus=64:mpiprocs=61:mem=500GB&lt;br /&gt;
&lt;br /&gt;
Requesting nodes that can satisfy those resources, two nodes with these attributes must exist. We have multiple nodes with 512GB in the epyc1 queue, meaning that this job will run on two machines that have at least the required amount of memory installed (on each node). The job will be queued until two machines like this will be available. If no machines with these resources exist, the job will stay in the queue forever. Therefore, you have to craft the submission string carefully.&lt;br /&gt;
&lt;br /&gt;
To accommodate high memory jobs, the nodes have been assigned priorities for assignment. Low memory jobs have the highest priority and will be assigned to nodes that can accommodate the request. High memory nodes have the lowest priority, meaning that they are the last ones given out to users. This makes it more likely that a high memory job can be run soon when the cluster is moderately loaded with jobs.&lt;br /&gt;
&lt;br /&gt;
StarCCM+ will always use the Intel MPI fabric. Other MPI versions do not work, even when selected on the command line.&lt;br /&gt;
&lt;br /&gt;
==OpenFOAM on the ARROW Cluster==&lt;br /&gt;
&lt;br /&gt;
===Currently Available OpenFOAM  Versions===&lt;br /&gt;
&lt;br /&gt;
As of late 2025, we have the following versions of OpenFOAM available on the cluster:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
$ module avail openfoam&lt;br /&gt;
------------ /shared/apps/modulefiles ------------&lt;br /&gt;
openfoam/9   openfoam/13      openfoam/v2312  &lt;br /&gt;
openfoam/10  openfoam/13-amd  openfoam/v2406  &lt;br /&gt;
openfoam/11  openfoam/v2212   &lt;br /&gt;
openfoam/12  openfoam/v2306&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
Contact us if you encounter problems; there can be various reasons why OpenFOAM may have trouble on certain hardware or when compiling dynamic code. When loading OpenFOAM modules, a number of dependencies will be automatically loaded for you, and you don&#039;t have to load those yourself. For example:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
$ module load openfoam/13&lt;br /&gt;
Loading openfoam/13&lt;br /&gt;
  Loading requirement: intel/2024.2.0/mpi/2021.13 gcc/gcc-12.1.0&lt;br /&gt;
&lt;br /&gt;
$ module list&lt;br /&gt;
Currently Loaded Modulefiles:&lt;br /&gt;
 1) intel/2024.2.0/mpi/2021.13   2) gcc/gcc-12.1.0   3) openfoam/13&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
In this case, OpenFOAM 13 loads the Intel 2024 MPI module, and loads the GCC compiler 12.1. OpenFOAM was compiled from source, and has been compiled specifically with that compiler and MPI version, so it make little sense to use other compilers or MPI libraries.&lt;br /&gt;
&lt;br /&gt;
Note: We have found a problem with running the Intel 2024 MPI library in the amd64 queue. Therefore, we have a modified module that uses the Intel 2022 library (I know -- Intel 2022 gives you the 2021 MPI libraries, but that is the way Intel distributes this software):&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
$ module load openfoam/13-amd &lt;br /&gt;
Loading mpi version 2021.7.0&lt;br /&gt;
Loading openfoam/13-amd&lt;br /&gt;
  Loading requirement: intel/2022.2.0/mpi/2021.7.0 gcc/gcc-12.1.0&lt;br /&gt;
&lt;br /&gt;
$ module list&lt;br /&gt;
Currently Loaded Modulefiles:&lt;br /&gt;
 1) intel/2022.2.0/mpi/2021.7.0   2) gcc/gcc-12.1.0   3) openfoam/13-amd&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
If you are compiling OpenFOAM yourself, the modules are of little help. You would need to select the appropriate MPI version and compiler before doing so, and then consistently load them before running your OpenFOAM executables. Within the &amp;quot;etc/bashrc&amp;quot; file in the source code tree, you want to set the MPI library to INTELMPI. As usual with self-compiled versions of OpenFOAM, you would &amp;quot;source etc/bashrc&amp;quot; to set up your personal environment to run your home-brew version of OpenFOAM. Contact us if you need to learn more about compiling OpenFOAM on the system.&lt;br /&gt;
&lt;br /&gt;
==Additional Software Applications and Libraries==&lt;br /&gt;
&lt;br /&gt;
===Loadable GCC Compiler Versions===&lt;br /&gt;
&lt;br /&gt;
The Rocky 9.6 operating system uses the GCC 11.5 compiler. That should be sufficient for most users when compiling your own applications. In case you need to use either a more up-to-date compiler, or if you need an older compiler for compatibility, we make the following versions available as loadable modules.&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
$ module avail gcc&lt;br /&gt;
------------ /shared/apps/modulefiles ------------&lt;br /&gt;
gcc/gcc-4.9.4  gcc/gcc-7.5.0  gcc/gcc-10.3.0  &lt;br /&gt;
gcc/gcc-5.5.0  gcc/gcc-8.5.0  gcc/gcc-11.3.0  &lt;br /&gt;
gcc/gcc-6.5.0  gcc/gcc-9.5.0  gcc/gcc-12.1.0&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
Additional versions can be created and made available as modules as well. If you need a specific version that is not currently available, please ask us to compiler and install it. If necessary, we may be able to provide access to other compilers, for example LLVM. We do not provide access to proprietary compilers at this time.&lt;br /&gt;
&lt;br /&gt;
===MPI Libraries and Runtimes===&lt;br /&gt;
&lt;br /&gt;
While we seem to have a variety of MPI versions and flavors available to users, the only MPI versions that allow us to run software over Infiniband are the Intel MPI libraries. Some of the installed alternatives are likely to fail, or will have a set of environment variables that have to be set. All major engineering software packages that we offer are pre-configured with specific MPI versions and settings that have been tested and/or provided by the vendors.&lt;br /&gt;
&lt;br /&gt;
Note: Some MPI libraries may seem to work. They may allow your MPI application to run. But inter-process network communication may travel through the rather slow and high-latency Ethernet fabric, making MPI applications very ineffective and are probably not worth while.&lt;br /&gt;
&lt;br /&gt;
===MatLab Runtimes===&lt;br /&gt;
&lt;br /&gt;
We can install MatLAB run time libraries as needed and have them available as loadable modules. Recently, we had a problem with MatLAB run time libraries being identified as security vulnerabilities. Contact us if you need them installed for one of your projects.&lt;br /&gt;
&lt;br /&gt;
===Anaconda and variants (miniconda etc)===&lt;br /&gt;
&lt;br /&gt;
Our current practice is to have users download and install their own versions of Anaconda and its variants in their own home directories. This allows for maximum flexibility when it comes to installable software modules, and users can maintain the installation, upgrades, and maintenance themselves. If you encounter issues, please contact us. One known side effect of Anaconda installations is a performance hit when starting your software, e.g. python scripts may take 30 seconds or more to execute. This is an artefact caused by the Lustre file system, which has been designed for large files accessible from many machines simultaneously. Performance on reading many small files has not been considered and is fairly poor. Again, contact us and we will design a solution for you as needed.&lt;/div&gt;</summary>
		<author><name>Ley</name></author>
	</entry>
	<entry>
		<id>https://wiki.anl.gov/wiki_tracc/index.php?title=Job_Submission_and_Monitoring&amp;diff=2487</id>
		<title>Job Submission and Monitoring</title>
		<link rel="alternate" type="text/html" href="https://wiki.anl.gov/wiki_tracc/index.php?title=Job_Submission_and_Monitoring&amp;diff=2487"/>
		<updated>2026-02-23T17:17:19Z</updated>

		<summary type="html">&lt;p&gt;Ley: /* Submitting an LS-Dyna Job */&lt;/p&gt;
&lt;hr /&gt;
&lt;div&gt;{| align=&amp;quot;right&amp;quot;&lt;br /&gt;
| __TOC__&lt;br /&gt;
|}&lt;br /&gt;
==Resource Summary View==&lt;br /&gt;
&lt;br /&gt;
To get started, users can query the overall status of resources on the cluster. The &#039;&#039;&#039;&amp;quot;qsum&amp;quot;&#039;&#039;&#039; script will list all queues and nodes, as well as how many are offline, down, free, or assigned to users. This is a script developed by our team, and may need to be updated if something goes wrong. Please contact us if you experience any problems.&lt;br /&gt;
&lt;br /&gt;
Each queue groups a number of nodes together based on their hardware and software configurations. Nodes can be part of more than one queue, and there are other complex details that we are ignoring here for the purpose of keeping it simple.&lt;br /&gt;
&lt;br /&gt;
===Queues===&lt;br /&gt;
&lt;br /&gt;
Here is a very brief summary of what each of the queues is, and how to use them efficiently:&lt;br /&gt;
&lt;br /&gt;
; a4000: This is a queue that has three 16-core CPU machines, each of which is furthermore equipped with three A4000 GPUs. That makes a total of 9 A4000 GPUs available to users. Neither the GPUs nor the processors are particularly powerful these days. The machines have 500GB of memory though, which makes for a good platform for experimenting with GPU capabilities.&lt;br /&gt;
; a6000: This is a queue that has only one 64-core CPU machines, and is equipped with four A6000 GPUs. The system can be upgraded to 8 A6000 GPUs if needed. This is a decent GPU machine that can take a solid workload these days. The machine has 750GB of memory, which makes for a good production platform.&lt;br /&gt;
; amd16: This is a queue with many of our older AMD-based 16-core machines, each of which has 30GB of memory. While individual machines are a bit outdated, they are all interconnected with Infiniband and can provide a solid production workload in multi-nodes jobs over MPI without blocking the more current (and thus expensive) systems.&lt;br /&gt;
; epyc1/epyc2: These are 2 separate queues with slightly different performance characteristics. Each of the groups is interconnected with Infiniband to provide a platform for large and demanding software packages, such as LS-Dyna and StarCCM+. They have between 250GB and 500GB of memory. Because licenses for these software packages are very expensive, they should use these two queues for making optimum use of limited core licenses available to each package.&lt;br /&gt;
; xeon28: This is a set of intermediate machines with 28 cores and 64GB of memory. They can be used for a variety of purposes, including MPI jobs and single node application software.&lt;br /&gt;
; virtual: This is a set of nodes without MPI capabilities. They are virtual machines with 32GB each. They can be used for higher demand applications that would interfere with the login nodes, and therefore with other users of these login machines. A user would submit interactive jobs to individual virtual machines and avoid any significant load on login nodes.&lt;br /&gt;
&lt;br /&gt;
===The Queue Summary Script (qsum)===&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
$ qsum&lt;br /&gt;
=============== a4000 ==========================================================&lt;br /&gt;
Queue: &amp;quot;a4000&amp;quot; / nodes: 3 / down: 0 / offline: 0 / busy: 0 / available: 3&lt;br /&gt;
      AVAILABLE (3): g001, g002, g003&lt;br /&gt;
=============== a6000 ==========================================================&lt;br /&gt;
Queue: &amp;quot;a6000&amp;quot; / nodes: 1 / down: 0 / offline: 0 / busy: 0 / available: 1&lt;br /&gt;
      AVAILABLE (1): lambda01&lt;br /&gt;
=============== amd16 ==========================================================&lt;br /&gt;
Queue: &amp;quot;amd16&amp;quot; / nodes: 33 / down: 2 / offline: 0 / busy: 2 / available: 29&lt;br /&gt;
           DOWN (2): n017, n030&lt;br /&gt;
            ley (2): n001, n002&lt;br /&gt;
     AVAILABLE (29): n003, n004, n005, n006, n007, n008, n009, n010, n011, n012&lt;br /&gt;
                     n013, n014, n015, n016, n018, n019, n020, n021, n022, n023&lt;br /&gt;
                     n024, n025, n026, n027, n028, n029, n031, n032, n039&lt;br /&gt;
=============== epyc1 ==========================================================&lt;br /&gt;
Queue: &amp;quot;epyc1&amp;quot; / nodes: 1 / down: 0 / offline: 0 / busy: 0 / available: 1&lt;br /&gt;
      AVAILABLE (1): a027&lt;br /&gt;
=============== epyc2 ==========================================================&lt;br /&gt;
Queue: &amp;quot;epyc2&amp;quot; / nodes: 20 / down: 0 / offline: 0 / busy: 5 / available: 15&lt;br /&gt;
            ley (2): a030, a031&lt;br /&gt;
         msitek (3): a028, a029, a032&lt;br /&gt;
     AVAILABLE (15): a033, a034, a035, a036, a037, a038, a039, a040, a041, a042&lt;br /&gt;
                     a043, a044, a045, a046, a047&lt;br /&gt;
=============== virtual ========================================================&lt;br /&gt;
Queue: &amp;quot;virtual&amp;quot; / nodes: 6 / down: 0 / offline: 0 / busy: 0 / available: 6&lt;br /&gt;
      AVAILABLE (6): v001, v002, v003, v004, v005, v006&lt;br /&gt;
=============== xeon28 =========================================================&lt;br /&gt;
Queue: &amp;quot;xeon28&amp;quot; / nodes: 12 / down: 0 / offline: 0 / busy: 0 / available: 12&lt;br /&gt;
     AVAILABLE (12): p001, p002, p003, p004, p005, p006, p007, p008, p009, p010&lt;br /&gt;
                     p011, p012&lt;br /&gt;
================================================================================&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
==Queue Status and Monitoring Jobs==&lt;br /&gt;
&lt;br /&gt;
===qstat===&lt;br /&gt;
&lt;br /&gt;
To find out out about the status of all running jobs on the cluster you can use the &#039;&#039;&#039;&amp;quot;qstat&amp;quot;&#039;&#039;&#039; command. Here is an example:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
$ qstat&lt;br /&gt;
&lt;br /&gt;
Nov 20 18:30 ley@login3:Plots$ qstat&lt;br /&gt;
Job id            Name             User              Time Use S Queue&lt;br /&gt;
----------------  ---------------- ----------------  -------- - -----&lt;br /&gt;
3023.pbs          STDIN            msitek            4144:14* R epyc2           &lt;br /&gt;
3029.pbs          STDIN            ley               76:46:53 R epyc2           &lt;br /&gt;
3032.pbs          STDIN            msitek            2879:52* R epyc2           &lt;br /&gt;
3033.pbs          STDIN            msitek            3687:29* R epyc2           &lt;br /&gt;
3048.pbs          foo.sh           james.cook               0 Q amd16           &lt;br /&gt;
3060.pbs          of13.sh          ley               310:47:* R epyc2           &lt;br /&gt;
3061.pbs          of13.sh          ley               308:37:* R epyc2           &lt;br /&gt;
3062.pbs          of13.sh          ley               308:02:* R epyc2           &lt;br /&gt;
3063.pbs          of13.sh          ley               308:15:* R epyc2&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
The first column shows the &#039;&#039;&#039;job id&#039;&#039;&#039;, a unique identifier for all jobs ever submitted to the cluster. This job id is important when killing jobs, or for other actions you may need to take.&lt;br /&gt;
&lt;br /&gt;
The next column shows the name of the job script. If the column shows &#039;&#039;&#039;STDIN&#039;&#039;&#039;, it means that this is an &#039;&#039;&#039;interactive job&#039;&#039;&#039; where a user can enter commands in a terminal window. This is particularly useful for model and software development task where the application has to be started and killed repeatedly.&lt;br /&gt;
&lt;br /&gt;
The owner of the job is shown next. These are the user names of the various people using the cluster.&lt;br /&gt;
&lt;br /&gt;
The last three columns indicate the current run time of the job, whether it is running (R) or waiting (Q) for execution. The last entry shows the queue in which the job is running.&lt;br /&gt;
&lt;br /&gt;
===qstat -an1===&lt;br /&gt;
&lt;br /&gt;
Adding a few options gives much more detail about each jobs.&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
qstat -an1&lt;br /&gt;
&lt;br /&gt;
Nov 20 13:09 ley@login3:Plots$ qstat -an1&lt;br /&gt;
&lt;br /&gt;
pbs: &lt;br /&gt;
                                                            Req&#039;d  Req&#039;d   Elap&lt;br /&gt;
Job ID          Username Queue    Jobname    SessID NDS TSK Memory Time  S Time&lt;br /&gt;
--------------- -------- -------- ---------- ------ --- --- ------ ----- - -----&lt;br /&gt;
3023.pbs        msitek   epyc2    STDIN      24360*   1  64  350gb 100:0 R 81:46 a028/0*64&lt;br /&gt;
3029.pbs        ley      epyc2    STDIN      21719*   2 128  100gb 200:0 R 72:31 a030/0*64+a031/0*64&lt;br /&gt;
3032.pbs        msitek   epyc2    STDIN      18102*   1  64  350gb 100:0 R 57:57 a029/0*64&lt;br /&gt;
3033.pbs        msitek   epyc2    STDIN      830486   1  64  350gb 100:0 R 57:53 a032/0*64&lt;br /&gt;
3048.pbs        james.c* amd16    foo.sh        --    1  28   30gb 01:00 Q   --   -- &lt;br /&gt;
3060.pbs        ley      epyc2    STDIN      763101   1  64  350gb 48:00 R 06:42 a033/0*64&lt;br /&gt;
3061.pbs        ley      epyc2    STDIN      763947   1  64  350gb 48:00 R 06:40 a034/0*64&lt;br /&gt;
3062.pbs        ley      epyc2    STDIN      761473   1  64  350gb 48:00 R 06:39 a035/0*64&lt;br /&gt;
3063.pbs        ley      epyc2    STDIN      766205   1  64  350gb 48:00 R 06:40 a036/0*64&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
In this table, you can see how many nodes and cores are being used by each job. For example, job 3029 of the user &amp;quot;ley&amp;quot; shows that it is running on 2 nodes using a total of 128 cores. In addition to the elapsed time, the table also show the reserved time for this job. This allow you to estimate when a job will be definitely finalized (or killed by the system if still running).&lt;br /&gt;
&lt;br /&gt;
The last column (without a header) is written because the option &#039;&#039;&#039;&amp;quot;-an1&amp;quot;&#039;&#039;&#039; was used. This useful to learn about which nodes are used by each job.&lt;br /&gt;
&lt;br /&gt;
===qstat -q===&lt;br /&gt;
&lt;br /&gt;
To learn more about the queues on the cluster, the &#039;&#039;&#039;&amp;quot;-q&amp;quot;&#039;&#039;&#039; option turns out to be useful.&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
$ qstat -q&lt;br /&gt;
&lt;br /&gt;
server: pbs&lt;br /&gt;
&lt;br /&gt;
Queue            Memory CPU Time Walltime Node   Run   Que   Lm  State&lt;br /&gt;
---------------- ------ -------- -------- ---- ----- ----- ----  -----&lt;br /&gt;
virtual            30gb    --       --       1     0     0   --   E R&lt;br /&gt;
a4000             500gb    --       --       1     0     0   --   E R&lt;br /&gt;
a6000             750gb    --       --       1     0     0   --   E R&lt;br /&gt;
xeon28             --      --       --       4     0     0   --   E R&lt;br /&gt;
amd16              --      --       --       8     0     1   --   E R&lt;br /&gt;
epyc2              --      --       --       2    14     0   --   E R&lt;br /&gt;
epyc1              --      --       --       2     0     0   --   E R&lt;br /&gt;
                                               ----- -----&lt;br /&gt;
                                                  14     1&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
For each queue, some basic values are displayed. The first three queues listed above have a default memory allocation as shown, and the column &#039;&#039;&#039;&amp;quot;Node&amp;quot;&#039;&#039;&#039; indicates the maximum number of nodes that can be asked for at job submission time. For example, you can request just one node for a job from the first three queues (because these are queues where MPI makes no sense). The &#039;&#039;&#039;&amp;quot;xeon28&amp;quot;&#039;&#039;&#039; queue also for a maximum of 4 nodes per MPI job. The &#039;&#039;&#039;&amp;quot;amd16&amp;quot;&#039;&#039;&#039; queue has a maximum of 8 nodes per job, and the &#039;&#039;&#039;&amp;quot;epyc1&amp;quot;&#039;&#039;&#039; and &#039;&#039;&#039;&amp;quot;epyc2&amp;quot;&#039;&#039;&#039; queues have maxima of two nodes per job. These limitations can be changed by the administrator as needed. As shown above, this will prevent inefficient resource requests.&lt;br /&gt;
&lt;br /&gt;
===qstat -f===&lt;br /&gt;
&lt;br /&gt;
The command &#039;&#039;&#039;&amp;quot;qstat -f -F json 3029&amp;quot;&#039;&#039;&#039; retrieves extremely detailed stats on the running job 3029. The result can be returned in JSON format to be ready for further processing (shown below).&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
$ qstat -f -F json 3029&lt;br /&gt;
{&lt;br /&gt;
    &amp;quot;timestamp&amp;quot;:1763705353,&lt;br /&gt;
    &amp;quot;pbs_version&amp;quot;:&amp;quot;23.06.06&amp;quot;,&lt;br /&gt;
    &amp;quot;pbs_server&amp;quot;:&amp;quot;pbs&amp;quot;,&lt;br /&gt;
    &amp;quot;Jobs&amp;quot;:{&lt;br /&gt;
        &amp;quot;3029.pbs&amp;quot;:{&lt;br /&gt;
            &amp;quot;Job_Name&amp;quot;:&amp;quot;STDIN&amp;quot;,&lt;br /&gt;
            &amp;quot;Job_Owner&amp;quot;:&amp;quot;ley@login4&amp;quot;,&lt;br /&gt;
            &amp;quot;resources_used&amp;quot;:{&lt;br /&gt;
                &amp;quot;cpupercent&amp;quot;:98,&lt;br /&gt;
                &amp;quot;cput&amp;quot;:&amp;quot;76:46:53&amp;quot;,&lt;br /&gt;
                &amp;quot;hpmem&amp;quot;:&amp;quot;0b&amp;quot;,&lt;br /&gt;
                &amp;quot;mem&amp;quot;:&amp;quot;52428800kb&amp;quot;,&lt;br /&gt;
                &amp;quot;ncpus&amp;quot;:128,&lt;br /&gt;
                &amp;quot;vmem&amp;quot;:&amp;quot;52428800kb&amp;quot;,&lt;br /&gt;
                &amp;quot;walltime&amp;quot;:&amp;quot;78:09:32&amp;quot;&lt;br /&gt;
            },&lt;br /&gt;
            &amp;quot;job_state&amp;quot;:&amp;quot;R&amp;quot;,&lt;br /&gt;
            &amp;quot;queue&amp;quot;:&amp;quot;epyc2&amp;quot;,&lt;br /&gt;
            &amp;quot;server&amp;quot;:&amp;quot;pbs&amp;quot;,&lt;br /&gt;
            &amp;quot;Checkpoint&amp;quot;:&amp;quot;u&amp;quot;,&lt;br /&gt;
            &amp;quot;ctime&amp;quot;:&amp;quot;Mon Nov 17 17:58:25 2025&amp;quot;,&lt;br /&gt;
            &amp;quot;Error_Path&amp;quot;:&amp;quot;/dev/pts/0&amp;quot;,&lt;br /&gt;
            &amp;quot;exec_host&amp;quot;:&amp;quot;a030/0*64+a031/0*64&amp;quot;,&lt;br /&gt;
            &amp;quot;exec_vnode&amp;quot;:&amp;quot;(a030:ncpus=64:mem=52428800kb)+(a031:ncpus=64:mem=52428800kb)&amp;quot;,&lt;br /&gt;
            &amp;quot;Hold_Types&amp;quot;:&amp;quot;n&amp;quot;,&lt;br /&gt;
            &amp;quot;interactive&amp;quot;:&amp;quot;True&amp;quot;,&lt;br /&gt;
            &amp;quot;Join_Path&amp;quot;:&amp;quot;n&amp;quot;,&lt;br /&gt;
            &amp;quot;Keep_Files&amp;quot;:&amp;quot;n&amp;quot;,&lt;br /&gt;
            &amp;quot;Mail_Points&amp;quot;:&amp;quot;a&amp;quot;,&lt;br /&gt;
            &amp;quot;mtime&amp;quot;:&amp;quot;Fri Nov 21 00:07:59 2025&amp;quot;,&lt;br /&gt;
            &amp;quot;Output_Path&amp;quot;:&amp;quot;/dev/pts/0&amp;quot;,&lt;br /&gt;
            &amp;quot;Priority&amp;quot;:0,&lt;br /&gt;
            &amp;quot;qtime&amp;quot;:&amp;quot;Mon Nov 17 17:58:25 2025&amp;quot;,&lt;br /&gt;
            &amp;quot;Rerunable&amp;quot;:&amp;quot;False&amp;quot;,&lt;br /&gt;
            &amp;quot;Resource_List&amp;quot;:{&lt;br /&gt;
                &amp;quot;mem&amp;quot;:&amp;quot;100gb&amp;quot;,&lt;br /&gt;
                &amp;quot;mpiprocs&amp;quot;:128,&lt;br /&gt;
                &amp;quot;ncpus&amp;quot;:128,&lt;br /&gt;
                &amp;quot;nodect&amp;quot;:2,&lt;br /&gt;
                &amp;quot;place&amp;quot;:&amp;quot;free&amp;quot;,&lt;br /&gt;
                &amp;quot;select&amp;quot;:&amp;quot;2:ncpus=64:mem=50gb:mpiprocs=64&amp;quot;,&lt;br /&gt;
                &amp;quot;walltime&amp;quot;:&amp;quot;200:00:00&amp;quot;&lt;br /&gt;
            },&lt;br /&gt;
            &amp;quot;stime&amp;quot;:&amp;quot;Mon Nov 17 17:58:25 2025&amp;quot;,&lt;br /&gt;
            &amp;quot;session_id&amp;quot;:2171964,&lt;br /&gt;
            &amp;quot;jobdir&amp;quot;:&amp;quot;/mnt/lustre/arrow/home/ley&amp;quot;,&lt;br /&gt;
            &amp;quot;substate&amp;quot;:42,&lt;br /&gt;
            &amp;quot;Variable_List&amp;quot;:{&lt;br /&gt;
                &amp;quot;PBS_O_HOME&amp;quot;:&amp;quot;/mnt/lustre/arrow/home/ley&amp;quot;,&lt;br /&gt;
                &amp;quot;PBS_O_LANG&amp;quot;:&amp;quot;en_US.UTF-8&amp;quot;,&lt;br /&gt;
                &amp;quot;PBS_O_LOGNAME&amp;quot;:&amp;quot;ley&amp;quot;,&lt;br /&gt;
                &amp;quot;PBS_O_PATH&amp;quot;:&amp;quot;/shared/apps/active/lstc/lsprepost/SP-4.5:/shared/apps/active/lstc/lsprepost/DP-4.3:/shared/bin:/usr/share/Modules/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/pbs/bin:/opt/thinlinc/bin:/opt/thinlinc/sbin:/mnt/lustre/arrow/home/ley/.local/bin:/mnt/lustre/arrow/home/ley/bin&amp;quot;,&lt;br /&gt;
                &amp;quot;PBS_O_MAIL&amp;quot;:&amp;quot;/var/spool/mail/ley&amp;quot;,&lt;br /&gt;
                &amp;quot;PBS_O_SHELL&amp;quot;:&amp;quot;/bin/bash&amp;quot;,&lt;br /&gt;
                &amp;quot;PBS_O_WORKDIR&amp;quot;:&amp;quot;/mnt/lustre/arrow/home/ley/Qualification/LS-Dyna/Rocky9/Seatbelt/Template&amp;quot;,&lt;br /&gt;
                &amp;quot;PBS_O_SYSTEM&amp;quot;:&amp;quot;Linux&amp;quot;,&lt;br /&gt;
                &amp;quot;PBS_O_QUEUE&amp;quot;:&amp;quot;epyc2&amp;quot;,&lt;br /&gt;
                &amp;quot;PBS_O_HOST&amp;quot;:&amp;quot;login4&amp;quot;&lt;br /&gt;
            },&lt;br /&gt;
            &amp;quot;comment&amp;quot;:&amp;quot;Job run at Mon Nov 17 at 17:58 on (a030:ncpus=64:mem=52428800kb)+(a031:ncpus=64:mem=52428800kb)&amp;quot;,&lt;br /&gt;
            &amp;quot;etime&amp;quot;:&amp;quot;Mon Nov 17 17:58:25 2025&amp;quot;,&lt;br /&gt;
            &amp;quot;run_count&amp;quot;:1,&lt;br /&gt;
            &amp;quot;Submit_arguments&amp;quot;:&amp;quot;-I -q epyc2 -l walltime=200:00:00,select=2:ncpus=64:mem=50gb:mpiprocs=64&amp;quot;,&lt;br /&gt;
            &amp;quot;project&amp;quot;:&amp;quot;_pbs_project_default&amp;quot;,&lt;br /&gt;
            &amp;quot;Submit_Host&amp;quot;:&amp;quot;login4&amp;quot;&lt;br /&gt;
        }&lt;br /&gt;
    }&lt;br /&gt;
}&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
===Manual pages for qstat===&lt;br /&gt;
&lt;br /&gt;
To learn more about the &#039;&#039;&#039;&amp;quot;qstat&amp;quot;&#039;&#039;&#039; command, you can use the command &#039;&#039;&#039;&amp;quot;man qstat&amp;quot;&#039;&#039;&#039;, which will print a lot of detailed information about the capabilities of this command.&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
$ man qstat&lt;br /&gt;
&lt;br /&gt;
qstat(1B)                                       PBS Professional                                       qstat(1B)&lt;br /&gt;
&lt;br /&gt;
NAME&lt;br /&gt;
       qstat - display status of PBS jobs, queues, or servers&lt;br /&gt;
&lt;br /&gt;
SYNOPSIS&lt;br /&gt;
       Displaying Job Status&lt;br /&gt;
              Default format:&lt;br /&gt;
              qstat [-E] [-J] [-p] [-t] [-w] [-x] [[&amp;lt;job ID&amp;gt; | &amp;lt;destination&amp;gt;] ...]&lt;br /&gt;
&lt;br /&gt;
              Long format:&lt;br /&gt;
              qstat -f [-F json | dsv [-D &amp;lt;delimiter&amp;gt;]] [-E] [-J] [-p] [-t] [-w]&lt;br /&gt;
                    [-x] [[&amp;lt;job ID&amp;gt; | &amp;lt;destination&amp;gt;] ...]&lt;br /&gt;
... &amp;lt;many more pages&amp;gt;&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
==Job Submission Basics==&lt;br /&gt;
&lt;br /&gt;
Jobs are submitted into the system using the &#039;&#039;&#039;&amp;quot;qsub&amp;quot;&#039;&#039;&#039; application. This application can take many different options and allows for a lot of different resource requests to tell the cluster what to do. We are running &#039;&#039;&#039;OpenPBS 23.06.06&#039;&#039;&#039; as our job scheduler. Here is a link to the User&#039;s Manual (of PBS PRO) if you want to explore gory details and capabilities. The User&#039;s Guide has about 240 pages, the Reference Guide has 500 pages, and the Big Book has 2500 pages. So there is a lot of information available. I also added job submission info for the LCRC cluster.&lt;br /&gt;
&lt;br /&gt;
* [https://argonne-lcrc.github.io/user-guides/running-jobs-at-lcrc/pbs-pro/ Argonne&#039;s LCRC pages on job submissions on their clusters]&lt;br /&gt;
* [https://help.altair.com/2022.1.0/PBS%20Professional/PBSUserGuide2022.1.pdf PBS Professional 2022.1 User&#039;s Guide]&lt;br /&gt;
* [https://help.altair.com/2022.1.0/PBS%20Professional/PBSReferenceGuide2022.1.pdf PBS Professional 2022.1 Reference Guide]&lt;br /&gt;
* [https://help.altair.com/2022.1.0/PBS%20Professional/PBS2022.1.pdf Altair PBS Professional 2022.1 Big Book]&lt;br /&gt;
&lt;br /&gt;
The User&#039;s Guide can be very helpful to clarify some of the concepts and capabilities, but it can be hard to find the specific information you may be looking for. Please understand that we are no longer running TORQUE and MAUI, so the syntax for job submission is distinctively different yet quite similar.&lt;br /&gt;
&lt;br /&gt;
The reference guide may be helpful to understand the complete syntax and full capabilities of the software.&lt;br /&gt;
&lt;br /&gt;
The big book is what I had to use when configuring OpenPBS earlier this year. This includes all the tricky details needed to make the system work smoothly for us. It&#039;s a bit scary to look at a PDF file that is 2500 pages long, but that is nothing compared to the StarCCM+ manuals.&lt;br /&gt;
&lt;br /&gt;
&amp;lt;BLOCKQUOTE&amp;gt;&lt;br /&gt;
[[File:Attention.jpg|25px]] &#039;&#039;&#039;IMPORTANT NOTE:&#039;&#039;&#039; &#039;&#039;The following sections are important to understand. They explain how jobs are submitted and then scjeduled for execution based on resources available and the specific need of the user.&#039;&#039;&lt;br /&gt;
&amp;lt;/BLOCKQUOTE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
The following sections explain the various tasks you may want to submit fir execution, ordered from simple to complex.&lt;br /&gt;
&lt;br /&gt;
* General Batch Jobs&lt;br /&gt;
** Requesting a Single Node for a Job&lt;br /&gt;
** Requesting Multiple Nodes for a Job&lt;br /&gt;
* Embedded Job Resource Requests&lt;br /&gt;
* Interactive Jobs&lt;br /&gt;
* Interactive Jobs with X-Windows GUI Applications&lt;br /&gt;
* Running Multiple Jobs on Single Nodes&lt;br /&gt;
* Running Jobs using GPUs&lt;br /&gt;
&lt;br /&gt;
===General Batch Jobs===&lt;br /&gt;
&lt;br /&gt;
Let&#039;s get started with a very basic usage of the system. Let&#039;s assume you have a simple application, and you want to execute it on a cluster node. Let&#039;s also assume that this is a very simple application, one that runs on one or a few cores, doesn&#039;t require any keyboard interaction with the user, doesn&#039;t need the user to see what&#039;s typically written to the screen, and writes its output to files. In this case, we can submit this application as a batch job, which will place it into an execution queue and process it as soon as a node becomes available.&lt;br /&gt;
&lt;br /&gt;
If the application requires more cores than a single node can provide, we can run the application over Infiniband with MPI message passing. In this case, we need to understand the concept of MPI applications a bit better. In both cases, we get started by creating a folder on the file system. Naming conventions are important so that you can distinguish the jobs by folder name.&lt;br /&gt;
&lt;br /&gt;
For both of the above scenarios, you would typically create a Bash shell script, and then submit the script into one of the queues for eventual execution.&lt;br /&gt;
&lt;br /&gt;
====Requesting a Single Node for a Job====&lt;br /&gt;
&lt;br /&gt;
Let&#039;s try something rather trivial to get used to the concept. Create yourself a folder, for example &#039;&#039;&#039;&amp;quot;myjobfolder&amp;quot;&#039;&#039;&#039;. Within that folder, create a job submission script. That script can have any name, but something short and simple may be best. Let&#039;s assume you create a file called &#039;&#039;&#039;&amp;quot;cluster.job&amp;quot;&#039;&#039;&#039;. The file doesn&#039;t have to have that extension. Any file name will do. But using the same filename for all of your jobs helps finding your way around the many files that will be created over time. The &#039;&#039;&#039;&amp;quot;cluster.job&amp;quot;&#039;&#039;&#039; file should look something like this:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
#!/bin/bash&lt;br /&gt;
#&lt;br /&gt;
# the following ensures that you will change into the directory where you are&lt;br /&gt;
# submitting the job from.&lt;br /&gt;
cd $PBS_O_WORKDIR&lt;br /&gt;
#&lt;br /&gt;
# now we sleep for 60 seconds and waste time. This is a placeholder for your application,&lt;br /&gt;
# which would be doing useful work for you.&lt;br /&gt;
sleep 60&lt;br /&gt;
#&lt;br /&gt;
# and after doing things, we may want to write something into a file to show that&lt;br /&gt;
# our jobs is done.&lt;br /&gt;
echo `date` &amp;gt; info.log&lt;br /&gt;
#&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
This can be submitted without detailed resource specifications (except for the walltime, which is be default 0:00:00):&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
$ qsub -q virtual -l walltime=1:00:00 cluster.job &lt;br /&gt;
3072.pbs&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
Wait a little, then check the status of running jobs:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
$ qstat -an1&lt;br /&gt;
&lt;br /&gt;
pbs: &lt;br /&gt;
                                                            Req&#039;d  Req&#039;d   Elap&lt;br /&gt;
Job ID          Username Queue    Jobname    SessID NDS TSK Memory Time  S Time&lt;br /&gt;
--------------- -------- -------- ---------- ------ --- --- ------ ----- - -----&lt;br /&gt;
3023.pbs        msitek   epyc2    STDIN      24360*   1  64  350gb 100:0 R 83:17 a028/0*64&lt;br /&gt;
3029.pbs        ley      epyc2    STDIN      21719*   2 128  100gb 200:0 R 74:00 a030/0*64+a031/0*64&lt;br /&gt;
3033.pbs        msitek   epyc2    STDIN      830486   1  64  350gb 100:0 R 59:23 a032/0*64&lt;br /&gt;
3048.pbs        james.c* amd16    foo.sh        --    1  28   30gb 01:00 Q   --   -- &lt;br /&gt;
3060.pbs        ley      epyc2    STDIN      763101   1  64  350gb 48:00 R 08:10 a033/0*64&lt;br /&gt;
3061.pbs        ley      epyc2    STDIN      763947   1  64  350gb 48:00 R 08:10 a034/0*64&lt;br /&gt;
3070.pbs        ley      epyc2    STDIN      766847   1  64  350gb 48:00 R 07:23 a042/0*64&lt;br /&gt;
3072.pbs        ley      virtual  cluster.j* 230230   1   4   30gb 01:00 E 00:01 v001/0*4&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
In this particular example, we are sending this job to the &#039;&#039;&#039;queue &amp;quot;virtual&amp;quot;&#039;&#039;&#039;. This queue, by default, allocates 30GB of memory to the job, and runs on 1 node with 4 cores. This is sufficient capacity to run quite a workload. When submitting a job to a single node, &#039;&#039;&#039;reasonable maximum allocations are automatically assigned&#039;&#039;&#039;, and the user doesn&#039;t have to worry about running out of memory or how many cores he will be using.&lt;br /&gt;
&lt;br /&gt;
The only required argument is the &#039;&#039;&#039;&amp;quot;walltime&amp;quot;&#039;&#039;&#039; argument. By default, the job will quit as soon as it is submitted. This indicates to the user that he forgot to provide the &#039;&#039;&#039;&amp;quot;walltime&amp;quot;&#039;&#039;&#039; argument.&lt;br /&gt;
&lt;br /&gt;
When the job disappears from the job list, it is done. At this point, you will find the file &amp;quot;info.log&amp;quot; in your job folder.&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
$ cat info.log &lt;br /&gt;
Thu Nov 20 08:00:31 PM CST 2025&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
====Requesting Multiple Nodes for a Job====&lt;br /&gt;
&lt;br /&gt;
To run jobs on multiple nodes, you will be likely &#039;&#039;&#039;executing jobs using MPI&#039;&#039;&#039;, the message passing interface. This establishes high-speed low-latency interconnections between the cores on one machine and the cores on the other machines. Data transfer does not require involvement of the cores themselves. Instead, the core tell the InfiniBand interconnect (and cores on the same node through shared memory) to transfer the data through RDMA, remoted direct memory access. The cores don&#039;t need to spend CPU cycles on copying data, but rather simply access the data once it has been copied by the Infiniband fabric. This makes for extremely efficient remote memory access, and message passing is used to coordinate data transfer between the cores no matter where they are located on any of the nodes.&lt;br /&gt;
&lt;br /&gt;
On our cluster, MPI-aware applications like &#039;&#039;&#039;OpenFOAM&#039;&#039;&#039;, &#039;&#039;&#039;StarCCM+&#039;&#039;&#039;, and &#039;&#039;&#039;LS-Dyna&#039;&#039;&#039; can be loaded as modules, which then automatically selects the most appropriate MPI library to use. The software applications have been tested to ensure that they work out-of-the box if a user selects any specific version of any of the applications.&lt;br /&gt;
&lt;br /&gt;
The following is a very trivial example for the MPI execution of a very simple executable, with one copy running on each core of the nodes allocated to  the job. It doesn&#039;t perform any real work and just wastes resources for a short time, but it illustrates how execution on the cores of various nodes works.&lt;br /&gt;
&lt;br /&gt;
Like in the previous section, we start with a simple job script that we submit to an appropriate queues. In this case, we pick a queue that has machines with Infiniband interfaces supporting efficient communications. Let&#039;s assume we edit a file with the name &#039;&#039;&#039;&amp;quot;parallel.job&amp;quot;&#039;&#039;&#039; like this:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
#!/bin/bash&lt;br /&gt;
#&lt;br /&gt;
# the following ensures that you will change into the directory where you are&lt;br /&gt;
# submitting the job from.&lt;br /&gt;
cd $PBS_O_WORKDIR&lt;br /&gt;
#&lt;br /&gt;
# to execute a simple command on all of the cores of all of the nodes allocated to the job,&lt;br /&gt;
# we need to make one of the MPI versions available. Let&#039;s use one of the most up-to-date&lt;br /&gt;
# MPI library available on the cluster&lt;br /&gt;
module load intel/2024.2.0/mpi/2021.13&lt;br /&gt;
#&lt;br /&gt;
# now we are apply a few settings that ensure that the MPI library will use the highest-performing&lt;br /&gt;
# Infiniband Interconnect, as well as a few options to tell MPI how to interface nodes with&lt;br /&gt;
# each other and which specific Infiniband adapter to use. This is complex and requires in-depth&lt;br /&gt;
# knowledge of the QLogic Infiniband adapters we are using. It is unlikely that you will ever have to&lt;br /&gt;
# deal with these options, because the &amp;quot;module load&amp;quot; command for the engineering applications we provide&lt;br /&gt;
# on ARROW will handle all those details transparently without the user needing to understand the details.&lt;br /&gt;
export I_MPI_HYDRA_BOOTSTRAP=ssh&lt;br /&gt;
export MPI_DEVICE=rdma:ofa-v2-ib0&lt;br /&gt;
export UCX_NET_DEVICES=qib0:1&lt;br /&gt;
#&lt;br /&gt;
# it doesn&#039;t make much sense, but in this example we are executing the OS command &amp;quot;uptime&amp;quot; on all cores&lt;br /&gt;
# of the nodes allocated to this job. The output from each core is written to the file info.log. We&lt;br /&gt;
# will find 56 lines of output in the file info.log, each created by the corresponding core executing&lt;br /&gt;
# the uptime command.&lt;br /&gt;
mpirun uptime &amp;gt; info.log&lt;br /&gt;
#&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
A good queue to test scripts is the &#039;&#039;&#039;&amp;quot;xeon28&amp;quot;&#039;&#039;&#039; queue. In the queue, we have 2 14-core Xeon processers per node, so that means that each node has 56 actual cores. We do not consider hyperthreading when doing parallel computing. 56 actual cores is what&#039;s being used here. The job submission will look like this:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
qsub -q xeon28 -l walltime=1:0:0 -l select=2:ncpus=28:mpiprocs=28:mem=60G parallel.job&lt;br /&gt;
        ^                  ^               ^       ^           ^      ^ ^ ^&lt;br /&gt;
        |                  |               |       |           |      | | + --- the name of the job script to execute&lt;br /&gt;
        |                  |               |       |           |      | + ----- don&#039;t forget to specify gigabytes&lt;br /&gt;
        |                  |               |       |           |      + ------- the amount of memory to request per node&lt;br /&gt;
        |                  |               |       |           + -------------- the number of MPI tasks per nodes&lt;br /&gt;
        |                  |               |       + -------------------------- the number of cores per node&lt;br /&gt;
        |                  |               + ---------------------------------- the number of nodes to select in the queue&lt;br /&gt;
        |                   + ------------------------------------------------- the requested time, in this case 1h&lt;br /&gt;
        + --------------------------------------------------------------------- the queue to be used for the job&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
At this point, the job has created a file &amp;quot;info.log&amp;quot; with 56 lines, one per core:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
 22:26:05 up 23 days,  9:28,  0 users,  load average: 0.00, 0.00, 0.00&lt;br /&gt;
 22:26:05 up 23 days,  9:28,  0 users,  load average: 0.00, 0.00, 0.00&lt;br /&gt;
 22:26:05 up 23 days,  9:28,  0 users,  load average: 0.00, 0.00, 0.00&lt;br /&gt;
 22:26:05 up 23 days,  9:28,  0 users,  load average: 0.00, 0.00, 0.00&lt;br /&gt;
 22:26:05 up 23 days,  9:28,  0 users,  load average: 0.00, 0.00, 0.00&lt;br /&gt;
 22:26:05 up 23 days,  9:28,  0 users,  load average: 0.00, 0.00, 0.00&lt;br /&gt;
 22:26:05 up 23 days,  9:53,  0 users,  load average: 0.06, 0.03, 0.00&lt;br /&gt;
 22:26:05 up 23 days,  9:53,  0 users,  load average: 0.06, 0.03, 0.00&lt;br /&gt;
 22:26:05 up 23 days,  9:53,  0 users,  load average: 0.06, 0.03, 0.00&lt;br /&gt;
...&lt;br /&gt;
&amp;lt;/PRE&amp;gt; &lt;br /&gt;
&lt;br /&gt;
In this simple example, the lines look all the same. Upon close examination through, you can find slightly different values for some of the lines. Some lines say that the machine is up for 23 days and 9:28, while others say 23 days and 9:53. Because all 28 cores of a node would see the same uptime of the server, half of the entries show one time stamp, and the other 28 cores show the other one. That demonstrates that the 56 processes have been running independently on 2 nodes.&lt;br /&gt;
&lt;br /&gt;
===Embedded Job Resource Requests===&lt;br /&gt;
&lt;br /&gt;
The job script can be modified to embed the resource requests in form of a series of &#039;&#039;&#039;#PBS&#039;&#039;&#039; statements at the beginning of the script file. This is a very common practice use at many HPC installations and job submission engines. Let&#039;s go back to the previous example where we run the script on two nodes in parallel. That is the &#039;&#039;&#039;&amp;quot;parallel.job&amp;quot;&#039;&#039;&#039; script file again:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
#!/bin/bash&lt;br /&gt;
#&lt;br /&gt;
#PBS -q xeon28&lt;br /&gt;
#PBS -l walltime=1:0:0&lt;br /&gt;
#PBS -l select=2:ncpus=28:mpiprocs=28:mem=60G&lt;br /&gt;
#&lt;br /&gt;
# the following ensures that you will change into the directory where you are&lt;br /&gt;
# submitting the job from.&lt;br /&gt;
cd $PBS_O_WORKDIR&lt;br /&gt;
#&lt;br /&gt;
# to execute a simple command on all of the cores of all of the nodes allocated to the job,&lt;br /&gt;
# we need to make one of the MPI versions available. Let&#039;s use one of the most up-to-date&lt;br /&gt;
# MPI library available on the cluster&lt;br /&gt;
module load intel/2024.2.0/mpi/2021.13&lt;br /&gt;
#&lt;br /&gt;
# now we are apply a few settings that ensure that the MPI library will use the highest-performing&lt;br /&gt;
# Infiniband Interconnect, as well as a few options to tell MPI how to interface nodes with&lt;br /&gt;
# each other and which specific Infiniband adapter to use. This is complex and requires in-depth&lt;br /&gt;
# knowledge of the QLogic Infiniband adapters we are using. It is unlikely that you will ever have to&lt;br /&gt;
# deal with these options, because the &amp;quot;module load&amp;quot; command for the engineering applications we provide&lt;br /&gt;
# on ARROW will handle all those details transparently without the user needing to understand the details.&lt;br /&gt;
export I_MPI_HYDRA_BOOTSTRAP=ssh&lt;br /&gt;
export MPI_DEVICE=rdma:ofa-v2-ib0&lt;br /&gt;
export UCX_NET_DEVICES=qib0:1&lt;br /&gt;
#&lt;br /&gt;
# it doesn&#039;t make much sense, but in this example we are executing the OS command &amp;quot;uptime&amp;quot; on all cores&lt;br /&gt;
# of the nodes allocated to this job. The output from each core is written to the file info.log. We&lt;br /&gt;
# will find 56 lines of output in the file info.log, each created by the corresponding core executing&lt;br /&gt;
# the uptime command.&lt;br /&gt;
mpirun uptime &amp;gt; info.log&lt;br /&gt;
#&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
If the resource requests are embedded within the file, they don&#039;t have to be specified on the command line any longer (the command line overrides the embedded specifications though). This may be convenient, because all the user has to do for job submission is the following:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
qsub parallel.job&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
Here is an example with more resource specifications and job settings that affect the behavior of the job:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
#!/bin/bash&lt;br /&gt;
#&lt;br /&gt;
#PBS -q xeon28&lt;br /&gt;
#PBS -l walltime=1:0:0&lt;br /&gt;
#PBS -l select=2:ncpus=28:mpiprocs=28:mem=60G&lt;br /&gt;
#PBS -A Account&lt;br /&gt;
#PBS -j oe&lt;br /&gt;
#PBS -N JobName&lt;br /&gt;
#PBS -e log.error&lt;br /&gt;
#PBS -o log.output&lt;br /&gt;
#PBS -m bae&lt;br /&gt;
#&lt;br /&gt;
...&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
I leave this to you as an exercise to figure out what the various options mean and how to specify them. There are many more, all documented in the PBS PRO manual (see above). Most of them are not terribly relevant and can be omitted.&lt;br /&gt;
&lt;br /&gt;
===Interactive Jobs===&lt;br /&gt;
&lt;br /&gt;
On ARROW, we don&#039;t restrict queues to be used only in batch mode. While &#039;&#039;&#039;batch mode&#039;&#039;&#039; is efficient for lining up a lot of work to be executed one after the other, ARROW has been designed to &#039;&#039;&#039;allow efficient model and software development in interactive mode&#039;&#039;&#039;. We have always ensured to have more computers than minimally needed to make it possible to dedicate resources to developers as needed, even if that means wasted CPU cycles. At times, we may ask you to limit the number of interactive jobs so that a large batch workload can be processed efficiently. This happens from time to time, and we have our users coordinate this with each other.&lt;br /&gt;
&lt;br /&gt;
Let&#039;s assume that you are developing an MPI application, or you are working on a complex &#039;&#039;&#039;OpenFOAM&#039;&#039;&#039; model that requires to start parallel processes over and over again just to find a bug and then fix it quickly. To do that, you can &#039;&#039;&#039;request an interactive job&#039;&#039;&#039; by adding the &#039;&#039;&#039;&amp;quot;-I&amp;quot;&#039;&#039;&#039; option to the job submission command (this is an uppercase I). Let&#039;s go to the parallel multi-node example from above:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
qsub -I -q xeon28 -l walltime=1:0:0 -l select=2:ncpus=28:mpiprocs=28:mem=60G&lt;br /&gt;
      ^    ^                  ^               ^       ^           ^      ^ ^&lt;br /&gt;
      |    |                  |               |       |           |      | + --- don&#039;t forget to specify gigabytes&lt;br /&gt;
      |    |                  |               |       |           |      + ----- the amount of memory to request per node&lt;br /&gt;
      |    |                  |               |       |           + ------------ the number of MPI tasks per nodes&lt;br /&gt;
      |    |                  |               |       + ------------------------ the number of cores per node&lt;br /&gt;
      |    |                  |               + -------------------------------- the number of nodes to select in the queue&lt;br /&gt;
      |    |                   + ----------------------------------------------- the requested time, in this case 1h&lt;br /&gt;
      |    + ------------------------------------------------------------------- the queue to be used for the job&lt;br /&gt;
      + ------------------------------------------------------------------------ request an interactive job &amp;lt;&amp;lt;===&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
When running interactive jobs with the &#039;&#039;&#039;&amp;quot;-I&amp;quot;&#039;&#039;&#039; parameter, we don&#039;t specify av job script at the end of the submission command. The interactive job will instead start (once the nodes are available) in interactive mode, meaning that the terminal session changes over from being a series of commands executed on the login server to being a series of commands being executed on the first node of the group of nodes that are allocated to the job. At this point, you can change to the desired working directory, but what you do with the allocated resources is entirely up to you. You can load modules, including MPI libraries, and then issue the commands for your application interactively and see how they execute. If you start an &#039;&#039;&#039;&amp;quot;mpirun&amp;quot;&#039;&#039;&#039;, the cores on your allocated secondary node will work as expected. There is no difference to batch mode, other than you having the ability to execute lines of commands at will.&lt;br /&gt;
&lt;br /&gt;
===Interactive Jobs with X-Windows GUI Applications===&lt;br /&gt;
&lt;br /&gt;
Interactive use can go further than that. With some of our software applications, like &#039;&#039;&#039;StarCCM+&#039;&#039;&#039;, you can run an &#039;&#039;&#039;interactive GUI application&#039;&#039;&#039; where you control the computational work from within the applications&#039; GUI. Within the GUI, you can control execution of the numerical solver that runs on as many cores as you requested, while being able to reconfigure the case through the GUI as well. Furthermore, you can visualize developing results on the fly by creating complex plots and visualizations.&lt;br /&gt;
&lt;br /&gt;
All that is need is an option &#039;&#039;&#039;&amp;quot;-X&amp;quot;&#039;&#039;&#039; being used as part of the job submission, like this:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
qsub -X -I -q xeon28 -l walltime=1:0:0 -l select=2:ncpus=28:mpiprocs=28:mem=60G&lt;br /&gt;
      ^  ^    ^                  ^               ^       ^           ^      ^ ^&lt;br /&gt;
      |  |    |                  |               |       |           |      | + --- don&#039;t forget to specify gigabytes&lt;br /&gt;
      |  |    |                  |               |       |           |      + ----- the amount of memory to request per node&lt;br /&gt;
      |  |    |                  |               |       |           + ------------ the number of MPI tasks per nodes&lt;br /&gt;
      |  |    |                  |               |       + ------------------------ the number of cores per node&lt;br /&gt;
      |  |    |                  |               + -------------------------------- the number of nodes to select in the queue&lt;br /&gt;
      |  |    |                   + ----------------------------------------------- the requested time, in this case 1h&lt;br /&gt;
      |  |    + ------------------------------------------------------------------- the queue to be used for the job&lt;br /&gt;
      |  + ------------------------------------------------------------------------ request an interactive job&lt;br /&gt;
      + --------------------------------------------------------------------------- request GUI capabilities &amp;lt;&amp;lt;===&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
===Running Multiple Jobs on Single Nodes===&lt;br /&gt;
&lt;br /&gt;
A feature that is new on ARROW is the ability to run multiple jobs on a single node. Let&#039;s assume that you are performing a sensitivity analysis on an existing model, and the model is simple enough to return results within a reasonable time on just a few cores of a higher end machine (maybe you are running SMP versions of &#039;&#039;&#039;LS-Dyna&#039;&#039;&#039;). Our high end machines have 64 cores, so lets assume you have an &#039;&#039;&#039;LS-Dyna&#039;&#039;&#039; model that runs well on 8 cores and doesn&#039;t use a whole lot of memory. In this case, you can submit individual jobs that request simply 8 cores and a fraction of the available memory available on the node, and all jobs execute independently from each other. Each job is fit into a slot where available. It is not very different from using whole nodes for everything. The important consideration is that each job is cleanly constrained into it&#039;s allotted resources using the &#039;&#039;&#039;CGROUPS&#039;&#039;&#039; functionality of modern operating systems. Because an abusive user cannot use more cores or more memory than allocated to his job, other users can safely run smaller jobs on the same node.&lt;br /&gt;
&lt;br /&gt;
Lets assume that we have a number of smaller jobs that we want to run on a single node in the &#039;&#039;&#039;&amp;quot;xeon28&amp;quot;&#039;&#039;&#039; queue. Each job would be submitted by using reduced resources that allow for sharing but that  guarantee that the jobs will be run successfully. In this case, you can &#039;&#039;&#039;submit many jobs&#039;&#039;&#039; in the following manner (with a job script for the small jobs, each of which can &#039;&#039;&#039;request varying resources&#039;&#039;&#039; if needed - some may want to run on 5 cores, others on 3):&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
#!/bin/bash&lt;br /&gt;
#&lt;br /&gt;
# the following ensures that you will change into the directory where you are&lt;br /&gt;
# submitting the job from.&lt;br /&gt;
cd $PBS_O_WORKDIR&lt;br /&gt;
#&lt;br /&gt;
# now we sleep for 300 seconds and waste time. This is a placeholder for your application,&lt;br /&gt;
# which would be doing useful work for you.&lt;br /&gt;
sleep 300&lt;br /&gt;
#&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
Now we submit a variety of these jobs (11 total in this example) to the &#039;&#039;&#039;&amp;quot;xeon28&amp;quot;&#039;&#039;&#039; queue for execution (note that the first few jobs request different amounts of memory and core counts):&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
qsub -q xeon28 -l walltime=1:0:0 -l select=1:ncpus=12:mpiprocs=12:mem=5G small.job &lt;br /&gt;
qsub -q xeon28 -l walltime=1:0:0 -l select=1:ncpus=10:mpiprocs=10:mem=7G small.job &lt;br /&gt;
qsub -q xeon28 -l walltime=1:0:0 -l select=1:ncpus=8:mpiprocs=8:mem=9G small.job &lt;br /&gt;
qsub -q xeon28 -l walltime=1:0:0 -l select=1:ncpus=16:mpiprocs=16:mem=20G small.job &lt;br /&gt;
qsub -q xeon28 -l walltime=1:0:0 -l select=1:ncpus=2:mpiprocs=2:mem=2G small.job &lt;br /&gt;
qsub -q xeon28 -l walltime=1:0:0 -l select=1:ncpus=2:mpiprocs=2:mem=2G small.job &lt;br /&gt;
qsub -q xeon28 -l walltime=1:0:0 -l select=1:ncpus=2:mpiprocs=2:mem=2G small.job &lt;br /&gt;
qsub -q xeon28 -l walltime=1:0:0 -l select=1:ncpus=2:mpiprocs=2:mem=2G small.job &lt;br /&gt;
qsub -q xeon28 -l walltime=1:0:0 -l select=1:ncpus=2:mpiprocs=2:mem=2G small.job &lt;br /&gt;
qsub -q xeon28 -l walltime=1:0:0 -l select=1:ncpus=2:mpiprocs=2:mem=2G small.job &lt;br /&gt;
qsub -q xeon28 -l walltime=1:0:0 -l select=1:ncpus=2:mpiprocs=2:mem=2G small.job &lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
They are now running in the order of submission, allocated on as few nodes in the &amp;quot;xeon28&amp;quot; queue as necessary. Only 2 nodes are being loaded quite heavily, and 4 more cores are in use on a third node.&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
Nov 20 23:34 ley@login3:myjobfolder$ qstat -an1&lt;br /&gt;
&lt;br /&gt;
pbs: &lt;br /&gt;
                                                            Req&#039;d  Req&#039;d   Elap&lt;br /&gt;
Job ID          Username Queue    Jobname    SessID NDS TSK Memory Time  S Time&lt;br /&gt;
--------------- -------- -------- ---------- ------ --- --- ------ ----- - -----&lt;br /&gt;
&lt;br /&gt;
3082.pbs        ley      xeon28   small.job  813221   1  12    5gb 01:00 R 00:01 p001/0*12&lt;br /&gt;
3083.pbs        ley      xeon28   small.job  813288   1  10    7gb 01:00 R 00:01 p001/1*10&lt;br /&gt;
3084.pbs        ley      xeon28   small.job  671792   1   8    9gb 01:00 R 00:01 p002/0*8&lt;br /&gt;
3085.pbs        ley      xeon28   small.job  671845   1  16   20gb 01:00 R 00:01 p002/1*16&lt;br /&gt;
3086.pbs        ley      xeon28   small.job  813361   1   2    2gb 01:00 R 00:00 p001/2*2&lt;br /&gt;
3087.pbs        ley      xeon28   small.job  813413   1   2    2gb 01:00 R 00:00 p001/3*2&lt;br /&gt;
3088.pbs        ley      xeon28   small.job  813464   1   2    2gb 01:00 R 00:00 p001/4*2&lt;br /&gt;
3089.pbs        ley      xeon28   small.job  671912   1   2    2gb 01:00 R 00:00 p002/2*2&lt;br /&gt;
3090.pbs        ley      xeon28   small.job  671969   1   2    2gb 01:00 R 00:00 p002/3*2&lt;br /&gt;
3091.pbs        ley      xeon28   small.job  632092   1   2    2gb 01:00 R 00:00 p003/0*2&lt;br /&gt;
3092.pbs        ley      xeon28   small.job  632100   1   2    2gb 01:00 R 00:00 p003/1*2&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
This is a particularly effective strategy to run concurrently many cases that don&#039;t scale well beyond a few cores. When running them on fewer cores but many of them at the same time, the overall processing rate will be much higher than executing them the traditional way.&lt;br /&gt;
&lt;br /&gt;
===Running Jobs using GPUs===&lt;br /&gt;
&lt;br /&gt;
The principle of running multiple jobs on a single node becomes particularly important when using servers equipped with &#039;&#039;&#039;GPUs&#039;&#039;&#039; for &#039;&#039;&#039;ML/AI&#039;&#039;&#039; applications. The cluster doesn&#039;t have a whole lot of &#039;&#039;&#039;GPUs&#039;&#039;&#039; at this point. We have three machines with three &#039;&#039;&#039;A4000&#039;&#039;&#039; GOUs, a &#039;&#039;&#039;total of 9 A4000 GPUs&#039;&#039;&#039;. Then we have a much more powerful single machine with our &#039;&#039;&#039;four A6000 GPUs&#039;&#039;&#039;.&lt;br /&gt;
&lt;br /&gt;
Using multiple GPUs in a single application is still something where the software has to be designed with hardware in mind. GPUs have several methods of communicating with each other, e.g. very fast &#039;&#039;&#039;NVLINK&#039;&#039;&#039; between pairs of GPUs, GPUs being directly connected to one of the two CPUs in the system and thus being able to communicate faster, and GPUs that have to jump between processors when communicating, and then the whole issue of having to go possibly through PCIe bridges.&lt;br /&gt;
&lt;br /&gt;
On our system, we are providing the ability to &#039;&#039;&#039;work mostly with individual GPUs&#039;&#039;&#039;. Users can also reserve entire nodes and develop or run applications that are adapted to that hardware, including several GPUs installed on that node. One thing we do not provide is the ability of GPU to GPU communication between nodes. Thus, a job cannot request more than one GPU node at a time.&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
qsub -q a4000 -I -l walltime=1:0:0 -l select=1:ncpus=5:mem=150G:ngpus=1&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
With these specifications, three single GPU jobs can run on a single server. Each job sees only one of the reserved GPU.&lt;br /&gt;
&lt;br /&gt;
To run a massive GPU job on 64 cores with 4 &#039;&#039;&#039;A6000 GPUs&#039;&#039;&#039;, submit the job like this:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
qsub -q a6000 -I -l walltime=1:0:0 -l select=1:ncpus=64:mem=725G:ngpus=4&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
===Manual pages for qsub===&lt;br /&gt;
&lt;br /&gt;
To learn more about the &amp;quot;qsub&amp;quot; command, you can use the command &amp;quot;man qsub&amp;quot;, which will print a lot of detailed information about the capabilities of this command.&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
$ man qsub&lt;br /&gt;
&lt;br /&gt;
qsub(1B)                                               PBS Professional                                              qsub(1B)&lt;br /&gt;
&lt;br /&gt;
NAME&lt;br /&gt;
       qsub - submit a job to PBS&lt;br /&gt;
&lt;br /&gt;
SYNOPSIS&lt;br /&gt;
       qsub [-a &amp;lt;date and time&amp;gt;] [-A &amp;lt;account string&amp;gt;] [-c &amp;lt;checkpoint spec&amp;gt;]&lt;br /&gt;
            [-C &amp;lt;directive prefix&amp;gt;] [-e &amp;lt;path&amp;gt;] [-f] [-h]&lt;br /&gt;
            [-I [-G [-- &amp;lt;GUI application/script&amp;gt;]] | [-X]] [-j &amp;lt;join&amp;gt;]&lt;br /&gt;
            [-J &amp;lt;range&amp;gt; [%&amp;lt;max subjobs]] [-k &amp;lt;discard&amp;gt;] [-l &amp;lt;resource list&amp;gt;]&lt;br /&gt;
            [-m &amp;lt;mail events&amp;gt;] [-M &amp;lt;user list&amp;gt;] [-N &amp;lt;name&amp;gt;] [-o &amp;lt;path&amp;gt;]&lt;br /&gt;
            [-p &amp;lt;priority&amp;gt;] [-P &amp;lt;project&amp;gt;] [-q &amp;lt;destination&amp;gt;] [-r &amp;lt;y|n&amp;gt;]&lt;br /&gt;
            [-R &amp;lt;remove options&amp;gt;] [-S &amp;lt;path list&amp;gt;] [-u &amp;lt;user list&amp;gt;]&lt;br /&gt;
            [-v &amp;lt;variable list&amp;gt;] [-V] [-W &amp;lt;additional attributes&amp;gt;] [-z]&lt;br /&gt;
            [- | &amp;lt;script&amp;gt; | -- &amp;lt;executable&amp;gt; [&amp;lt;arguments to executable&amp;gt;]]&lt;br /&gt;
       qsub --version&lt;br /&gt;
&lt;br /&gt;
DESCRIPTION&lt;br /&gt;
       You use the qsub command to submit a batch job to PBS.  Submitting a PBS job specifies a task, requests resources, and&lt;br /&gt;
       sets job attributes.&lt;br /&gt;
... &amp;lt;many more pages&amp;gt;&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
==LS-Dyna on the ARROW Cluster==&lt;br /&gt;
&lt;br /&gt;
===Currently Available LS-Dyna Versions===&lt;br /&gt;
&lt;br /&gt;
The following is a list of &#039;&#039;&#039;LS-Dyna versions&#039;&#039;&#039; available on &#039;&#039;&#039;ARROW&#039;&#039;&#039; after the latest reconfiguration of the system. As per LSTC/ANSYS, &#039;&#039;&#039;versions before 14.0.0 are not necessarily fully supported any longer&#039;&#039;&#039; because they are supposedly not compatible with modern operating systems and cannot be made to work reliably. We have tested the listed older versions of LS-Dyna and they have passed basic tests. They may not behave exactly as they did on the old CentOS 7 operating system, and time will show whether they can still be used or whether you will need to update your models and use a fully supported version.&lt;br /&gt;
&lt;br /&gt;
All versions are loaded using the &#039;&#039;&#039;&amp;quot;module load&amp;quot;&#039;&#039;&#039; command. Versions can be listed with the &#039;&#039;&#039;&amp;quot;module avail ls-dyna&amp;quot;&#039;&#039;&#039; command. To load one of the modules, use the following syntax:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
module load ls-dyna/14.2.0/mpi-d8-ifort190-avx512&lt;br /&gt;
            ^       ^      ^   ^  ^        ^&lt;br /&gt;
            |       |      |   |  |        + --- specify the extended instruction set needed for execution&lt;br /&gt;
            |       |      |   |  + ------------ load the version of the compiler that was used to create this &lt;br /&gt;
            |       |      |   + --------------- load the version that supports double precision variables&lt;br /&gt;
            |       |      + ------------------- load the MPP (MPI) version of LS-Dyna&lt;br /&gt;
            |       + -------------------------- load specifically version 14.2.0&lt;br /&gt;
            + ---------------------------------- load a version of LS-Dyna&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
The version string is composed of multiple elements to indicate variants in compilers and compiler options. Use the following guideline to choose an appropriate version to load:&lt;br /&gt;
&lt;br /&gt;
* &#039;&#039;&#039;&amp;quot;1&amp;quot;&#039;&#039;&#039; or &#039;&#039;&#039;&amp;quot;mpi&amp;quot;&#039;&#039;&#039; indicates whether this is a single node version of LS-Dyna (&#039;&#039;&#039;SMP&#039;&#039;&#039;) or whether this is a multi-node MPI version (&#039;&#039;&#039;MPP&#039;&#039;&#039;). All MPI versions use the &#039;&#039;&#039;IntelMPI 2022&#039;&#039;&#039; libraries which have been tested thoroughly on ARROW. MPI versions will use the Infiniband Network of ARROW for high-speed and low-latency inter-process communication using RDMA (remote direct memory access).&lt;br /&gt;
* All LS-Dyna versions are available in either &#039;&#039;&#039;floating point&#039;&#039;&#039; or &#039;&#039;&#039;double precision variants&#039;&#039;&#039;. Floating point variants use 4 bytes to represent a value, and double precision variants use 8 bytes. There are pros and cons for choosing one over the other variant. With regards to computational efficiency, both perform nearly the same because all machines are equipped with 64-bit CPUs.&lt;br /&gt;
** &#039;&#039;&#039;&amp;quot;f4&amp;quot;&#039;&#039;&#039; floating point versions&lt;br /&gt;
*** Pros: These require significantly less memory to run. Results occupy less disk space, and can be transferred significantly faster into and out of ARROW.&lt;br /&gt;
*** Cons: The numerical resolution is limited to 7 significant digits, which is often undesirable when dealing with mathematical operations on small and large numbers at the same time.&lt;br /&gt;
** &#039;&#039;&#039;&amp;quot;r8&amp;quot;&#039;&#039;&#039; double precision versions&lt;br /&gt;
*** Pros: The numerical resolution is about twice the number of significant digits compare to &amp;quot;f4&amp;quot;, which helps when when dealing with mathematical operations on small and large numbers at the same time.&lt;br /&gt;
*** Cons: These require more memory to run. Results occupy more disk space, and it takes longer to transfer data into and out of ARROW.&lt;br /&gt;
* There are two more identifiers to choose from when it comes to the variants of the executables: the specific compiler used to create the executable and the specific processor instruction set required for running the executable.&lt;br /&gt;
** For modern versions of LS-Dyna, two compilers have been used by the developers to create LS-Dyna executables: the &#039;&#039;&#039;Intel Fortran Compiler&#039;&#039;&#039; and the &#039;&#039;&#039;AOCC (AMD Optimizing C/C++ and Fortran) compiler&#039;&#039;&#039;. Both variants of the software are supported on ARROW. This gives users the opportunity to choose an alternate variant of the same LS-Dyna version when running into bugs or crashes.&lt;br /&gt;
** The variants based on the various instruction set extensions (&#039;&#039;&#039;SSE2&#039;&#039;&#039;, &#039;&#039;&#039;AVX2&#039;&#039;&#039;, &#039;&#039;&#039;AVX512&#039;&#039;&#039;, and so on) gives users even more options when choosing an alternate LS-Dyna variant of the same version when running into bugs or crashes. These instruction sets are mostly related to performance gains on specific processors. We have not performed thorough performance tests and cannot recommend specific versions right now.&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
$ module avail ls-dyna&lt;br /&gt;
--------------------------------------------- /shared/apps/modulefiles ---------------------------------------------&lt;br /&gt;
ls-dyna/09.3.1/1-d8-ifort131           ls-dyna/12.2.1/mpi-f4-ifort160-sse2    ls-dyna/14.2.0/mpi-f4-ifort190-avx512  &lt;br /&gt;
ls-dyna/09.3.1/1-f4-ifort131           ls-dyna/12.2.2/1-d8-ifort160-sse2      ls-dyna/14.2.0/mpi-f4-ifort190-sse2    &lt;br /&gt;
ls-dyna/09.3.1/mpi-d8-ifort131-avx2    ls-dyna/12.2.2/1-f4-ifort160-sse2      ls-dyna/15.0.2/1-d8-ifort190-sse2      &lt;br /&gt;
ls-dyna/09.3.1/mpi-d8-ifort131-avx512  ls-dyna/12.2.2/mpi-d8-aocc400-avx2     ls-dyna/15.0.2/1-f4-ifort190-sse2      &lt;br /&gt;
ls-dyna/09.3.1/mpi-f4-ifort131-avx2    ls-dyna/12.2.2/mpi-d8-ifort160-avx2    ls-dyna/15.0.2/mpi-d8-aocc400-avx2     &lt;br /&gt;
ls-dyna/09.3.1/mpi-f4-ifort131-avx512  ls-dyna/12.2.2/mpi-d8-ifort160-sse2    ls-dyna/15.0.2/mpi-d8-ifort190-avx2    &lt;br /&gt;
ls-dyna/10.2.0/1-d8-ifort160           ls-dyna/12.2.2/mpi-f4-aocc400-avx2     ls-dyna/15.0.2/mpi-d8-ifort190-avx512  &lt;br /&gt;
ls-dyna/10.2.0/1-f4-ifort160           ls-dyna/12.2.2/mpi-f4-ifort160-avx2    ls-dyna/15.0.2/mpi-d8-ifort190-sse2    &lt;br /&gt;
ls-dyna/11.0.0/1-d8-ifort160           ls-dyna/12.2.2/mpi-f4-ifort160-sse2    ls-dyna/15.0.2/mpi-f4-aocc400-avx2     &lt;br /&gt;
ls-dyna/11.0.0/1-f4-ifort160           ls-dyna/13.0.0/1-d8-ifort190           ls-dyna/15.0.2/mpi-f4-ifort190-avx2    &lt;br /&gt;
ls-dyna/11.1.0/1-d8-ifort160-sse2      ls-dyna/13.0.0/1-f4-ifort190           ls-dyna/15.0.2/mpi-f4-ifort190-avx512  &lt;br /&gt;
ls-dyna/11.1.0/1-f4-ifort160-sse2      ls-dyna/13.0.0/mpi-d8-ifort190-avx2    ls-dyna/15.0.2/mpi-f4-ifort190-sse2    &lt;br /&gt;
ls-dyna/11.2.0/1-d8-ifort160           ls-dyna/13.0.0/mpi-d8-ifort190-sse2    ls-dyna/16.0.0/1-d8-aocc420-avx2       &lt;br /&gt;
ls-dyna/11.2.0/1-f4-ifort160           ls-dyna/13.0.0/mpi-f4-ifort190-avx2    ls-dyna/16.0.0/1-d8-aocc420-avx512     &lt;br /&gt;
ls-dyna/11.2.0/mpi-f4-ifort160-avx2    ls-dyna/13.0.0/mpi-f4-ifort190-sse2    ls-dyna/16.0.0/1-d8-ifort190-sse2      &lt;br /&gt;
ls-dyna/11.2.0/mpi-f4-ifort160-sse2    ls-dyna/13.1.0/mpi-d8-aocc310-avx2     ls-dyna/16.0.0/1-f4-aocc420-avx2       &lt;br /&gt;
ls-dyna/11.2.1/1-d8-ifort160           ls-dyna/13.1.0/mpi-d8-ifort190-avx2    ls-dyna/16.0.0/1-f4-aocc420-avx512     &lt;br /&gt;
ls-dyna/11.2.1/1-f4-ifort160           ls-dyna/13.1.0/mpi-d8-ifort190-sse2    ls-dyna/16.0.0/1-f4-ifort190-sse2      &lt;br /&gt;
ls-dyna/11.2.1/mpi-d8-ifort160-avx2    ls-dyna/13.1.0/mpi-f4-aocc310-avx2     ls-dyna/16.0.0/mpi-d8-aocc420-avx2     &lt;br /&gt;
ls-dyna/11.2.1/mpi-d8-ifort160-sse2    ls-dyna/13.1.0/mpi-f4-ifort190-avx2    ls-dyna/16.0.0/mpi-d8-aocc420-avx512   &lt;br /&gt;
ls-dyna/11.2.1/mpi-f4-ifort160-avx2    ls-dyna/13.1.0/mpi-f4-ifort190-sse2    ls-dyna/16.0.0/mpi-d8-ifort190-avx2    &lt;br /&gt;
ls-dyna/11.2.1/mpi-f4-ifort160-sse2    ls-dyna/13.1.1/mpi-d8-ifort190-avx2    ls-dyna/16.0.0/mpi-d8-ifort190-avx512  &lt;br /&gt;
ls-dyna/11.2.2/mpi-d8-ifort160-avx2    ls-dyna/13.1.1/mpi-d8-ifort190-sse2    ls-dyna/16.0.0/mpi-d8-ifort190-sse2    &lt;br /&gt;
ls-dyna/11.2.2/mpi-d8-ifort160-sse2    ls-dyna/13.1.1/mpi-f4-ifort190-avx2    ls-dyna/16.0.0/mpi-f4-aocc420-avx2     &lt;br /&gt;
ls-dyna/11.2.2/mpi-f4-ifort160-avx2    ls-dyna/13.1.1/mpi-f4-ifort190-sse2    ls-dyna/16.0.0/mpi-f4-aocc420-avx512   &lt;br /&gt;
ls-dyna/11.2.2/mpi-f4-ifort160-sse2    ls-dyna/14.0.0/1-d8-ifort190           ls-dyna/16.0.0/mpi-f4-ifort190-avx2    &lt;br /&gt;
ls-dyna/12.1.0/1-d8-ifort160           ls-dyna/14.0.0/1-f4-ifort190           ls-dyna/16.0.0/mpi-f4-ifort190-avx512  &lt;br /&gt;
ls-dyna/12.1.0/1-f4-aocc310            ls-dyna/14.0.0/mpi-d8-aocc310-avx2     ls-dyna/16.0.0/mpi-f4-ifort190-sse2    &lt;br /&gt;
ls-dyna/12.1.0/1-f4-ifort160           ls-dyna/14.0.0/mpi-d8-ifort190-avx2    ls-dyna/16.1.0/mpi-d8-aocc420-avx2     &lt;br /&gt;
ls-dyna/12.1.0/mpi-d8-aocc310-avx2     ls-dyna/14.0.0/mpi-d8-ifort190-sse2    ls-dyna/16.1.0/mpi-d8-aocc420-avx512   &lt;br /&gt;
ls-dyna/12.1.0/mpi-d8-ifort160-avx2    ls-dyna/14.0.0/mpi-f4-ifort190-avx2    ls-dyna/16.1.0/mpi-d8-ifort190-avx2    &lt;br /&gt;
ls-dyna/12.1.0/mpi-d8-ifort160-sse2    ls-dyna/14.0.0/mpi-f4-ifort190-sse2    ls-dyna/16.1.0/mpi-d8-ifort190-avx512  &lt;br /&gt;
ls-dyna/12.1.0/mpi-f4-aocc310-avx2     ls-dyna/14.1.0/1-d8-ifort190-sse2      ls-dyna/16.1.0/mpi-d8-ifort190-sse2    &lt;br /&gt;
ls-dyna/12.1.0/mpi-f4-ifort160-avx2    ls-dyna/14.1.0/1-f4-ifort190-sse2      ls-dyna/16.1.0/mpi-f4-aocc420-avx2     &lt;br /&gt;
ls-dyna/12.1.0/mpi-f4-ifort160-sse2    ls-dyna/14.1.0/mpi-d8-aocc400-avx2     ls-dyna/16.1.0/mpi-f4-aocc420-avx512&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
===Submitting an LS-Dyna Job===&lt;br /&gt;
&lt;br /&gt;
&amp;lt;BLOCKQUOTE&amp;gt;&lt;br /&gt;
[[File:Attention.jpg|25px]] &#039;&#039;&#039;IMPORTANT NOTE:&#039;&#039;&#039; &#039;&#039;The job/queue manager can now track the number of LS-Dyna licenses given out to individual&lt;br /&gt;
jobs. At submission time, it is not possible to know what software a user may run. But by adding the clause &amp;quot;-l dynalic&amp;quot; at submission time,&lt;br /&gt;
the queue manager can calculate the total number of cores required and keep track of LS-Dyna licenses used by the job. When loading a version of LS-Dyna, a check will be performed, and LS-Dyna will be prevented from running if the &amp;quot;-l dynalic&amp;quot; clause was not used when submitting the job.&lt;br /&gt;
&lt;br /&gt;
&amp;lt;!--&lt;br /&gt;
&#039;&#039;The job/queue manager can track the number of LS-Dyna licenses to some degree. If all &#039;&#039;&#039;LS-Dyna users&#039;&#039;&#039; cooperate and use a script like the one shown below when submitting their jobs, the total number of concurrent &#039;&#039;&#039;LS-Dyna licenses&#039;&#039;&#039; will be tracked by the job manager correctly. That means that users can submit any number of LS-Dyna jobs, and jobs will only start when a sufficient number of licenses is available. This is managed by the &#039;&#039;&#039;&amp;quot;dynalic&amp;quot;&#039;&#039;&#039; resource at the end of the select statement. In this example, a 2-node job on 64-core nodes will need a total of &#039;&#039;&#039;&amp;quot;dynalic=128&amp;quot;&#039;&#039;&#039; licenses. This accounting breaks down when users don&#039;t use the &#039;&#039;&#039;&amp;quot;dynalic=XXX&amp;quot;&#039;&#039;&#039; statement, or when they don&#039;t calculate the number of licenses correctly. In that case, LS-Dyna jobs of all users are subject to sudden failure when LS-Dyna licenses run out. Please understand the importance of this specific setting in your job.&#039;&#039;&lt;br /&gt;
--&amp;gt;&lt;br /&gt;
&amp;lt;/BLOCKQUOTE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
Furthermore, careful consideration should be given with regards to choice of resources for an &#039;&#039;&#039;LS-Dyna job&#039;&#039;&#039;. With 64 cores available on a single node in the &#039;&#039;&#039;&amp;quot;epyc1&amp;quot;&#039;&#039;&#039; and &#039;&#039;&#039;&amp;quot;epyc2&amp;quot;&#039;&#039;&#039; queues, it may be counterproductive to run a job on two nodes instead of a single node. Users should run their jobs with different numbers of nodes and determine whether performance increases. It may well decrease when running a job on two or more nodes. The outcome of such tests will tell what the best allocation of resources will be.&lt;br /&gt;
&lt;br /&gt;
Most users use a job script like the following. All methods for job submission the the previous chapters apply as well, so there is a lot of flexibility:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
#!/bin/bash&lt;br /&gt;
#&lt;br /&gt;
#PBS -q epyc1&lt;br /&gt;
#PBS -l walltime=12:0:0&lt;br /&gt;
#PBS -l select=2:ncpus=64:mpiprocs=64:mem=225G,dynalic=128&lt;br /&gt;
#PBS -N JobName&lt;br /&gt;
#PBS -e log.error&lt;br /&gt;
#PBS -o log.output&lt;br /&gt;
#&lt;br /&gt;
cd $PBS_O_WORKDIR&lt;br /&gt;
#&lt;br /&gt;
module load ls-dyna/12.2.1/mpi-f4-ifort160-avx2&lt;br /&gt;
module load dynamore/current&lt;br /&gt;
#&lt;br /&gt;
mpirun ls-dyna i=main.k memory1=300m memory2=100m &amp;gt; dyna.log&lt;br /&gt;
#&lt;br /&gt;
# when using the Dynamore tools, you can start something like this at the end&lt;br /&gt;
DM.plotcprs.lnx -merge &amp;gt;&amp;gt; dyna.log&lt;br /&gt;
#&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
===LSTC Tools: LS-OPT and LS-PREPOST===&lt;br /&gt;
&lt;br /&gt;
For the new Rocky 9 cluster, I have not looked deeply into the ls-opt and ls-prepost packages that were installed. I noticed though that the LSTC server provided access to much newer versions of both software packages. If you would like to learn more or have a specific version in mind, I can easily download and install it for you.&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
$ module avail ls-opt&lt;br /&gt;
----------------------------------------------- /shared/apps/modulefiles ------------------------------------------------&lt;br /&gt;
ls-opt/5.1.1  ls-opt/6.0.0  ls-opt/7.0.0  ls-opt/7.0.2   ls-opt/2022R2  &lt;br /&gt;
ls-opt/5.2.1  ls-opt/6.1.0  ls-opt/7.0.1  ls-opt/2022R1  ls-opt/2023R1  &lt;br /&gt;
&lt;br /&gt;
$ module avail ls-prepost&lt;br /&gt;
----------------------------------------------- /shared/apps/modulefiles ------------------------------------------------&lt;br /&gt;
ls-prepost/4.5.10  ls-prepost/4.8.13  ls-prepost/4.8.30  ls-prepost/4.9.16  ls-prepost/4.10.7 &lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
===Dynamore Software===&lt;br /&gt;
&lt;br /&gt;
The Dynamore tools are available as a module:&lt;br /&gt;
&lt;br /&gt;
 module load dynamore/current&lt;br /&gt;
&lt;br /&gt;
We typically acquire a yearly license for the tools as we purchase licenses for LS-Dyna.&lt;br /&gt;
&lt;br /&gt;
===Vendor License File Installation===&lt;br /&gt;
&lt;br /&gt;
If you would like for us to install a vendor license for LS-Dyna models, please contact us for the required information. We can send you the general LS-Dyna license file to show the host ids for the license server. Using that information, your vendor should be able to create a vendor license file. Please send that file to us per Email or by other means.&lt;br /&gt;
&lt;br /&gt;
==StarCCM+ on the ARROW Cluster==&lt;br /&gt;
&lt;br /&gt;
===Currently Available StarCCM+ Versions===&lt;br /&gt;
&lt;br /&gt;
As of late 2025, we have the following versions of &#039;&#039;&#039;StarCCM+&#039;&#039;&#039; available on the cluster:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
$ module avail starccm&lt;br /&gt;
---------------------------- /shared/apps/modulefiles ----------------------------&lt;br /&gt;
starccm/15.02.007-R8  starccm/16.06.008-R8  starccm/18.06.006-R8  &lt;br /&gt;
starccm/15.02.009-R8  starccm/17.02.007-R8  starccm/19.02.009-R8  &lt;br /&gt;
starccm/15.04.008-R8  starccm/17.02.008-R8  starccm/20.04.007-R8  &lt;br /&gt;
starccm/15.06.008-R8  starccm/17.04.007-R8  starccm/20.06.007-R8  &lt;br /&gt;
starccm/16.02.008-R8  starccm/17.06.007-R8  &lt;br /&gt;
starccm/16.04.007-R8  starccm/18.04.008-R8&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
If using a &#039;&#039;&#039;single node&#039;&#039;&#039; for StarCCM+, job submission (for an interactive job) is simple and will use appropriate default settings:&lt;br /&gt;
&lt;br /&gt;
 qsub -I -X -q epyc1 -l walltime=20:00:00&lt;br /&gt;
&lt;br /&gt;
StarCCM+ can make use of the job scheduler attributes by automatically obtaining the number of cores and other resources from OpenPBS. In this case, the default number of cores and mpi processes for StarCCM+ are both 64 when using the epyc1 queue. So you can start your StarCCM+ run with:&lt;br /&gt;
&lt;br /&gt;
 module load starccm/15.02.007-R8 (or any other version)&lt;br /&gt;
 starccm+ -bs pbs&lt;br /&gt;
&lt;br /&gt;
In this case, there is no need for StarCCM+ to be told to run the case in parallel with the selected number of cores/mpiprocs.&lt;br /&gt;
&lt;br /&gt;
This can get a bit more complex when running on multiple nodes or when requesting high memory nodes. In that case you would use job submission parameters as shown below:&lt;br /&gt;
&lt;br /&gt;
 qsub -I -X -q epyc1 -l walltime=20:00:00,select=2:ncpus=64:mpiprocs=61:mem=500GB&lt;br /&gt;
&lt;br /&gt;
Requesting nodes that can satisfy those resources, two nodes with these attributes must exist. We have multiple nodes with 512GB in the epyc1 queue, meaning that this job will run on two machines that have at least the required amount of memory installed (on each node). The job will be queued until two machines like this will be available. If no machines with these resources exist, the job will stay in the queue forever. Therefore, you have to craft the submission string carefully.&lt;br /&gt;
&lt;br /&gt;
To accommodate high memory jobs, the nodes have been assigned priorities for assignment. Low memory jobs have the highest priority and will be assigned to nodes that can accommodate the request. High memory nodes have the lowest priority, meaning that they are the last ones given out to users. This makes it more likely that a high memory job can be run soon when the cluster is moderately loaded with jobs.&lt;br /&gt;
&lt;br /&gt;
StarCCM+ will always use the Intel MPI fabric. Other MPI versions do not work, even when selected on the command line.&lt;br /&gt;
&lt;br /&gt;
==OpenFOAM on the ARROW Cluster==&lt;br /&gt;
&lt;br /&gt;
===Currently Available OpenFOAM  Versions===&lt;br /&gt;
&lt;br /&gt;
As of late 2025, we have the following versions of OpenFOAM available on the cluster:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
$ module avail openfoam&lt;br /&gt;
------------ /shared/apps/modulefiles ------------&lt;br /&gt;
openfoam/9   openfoam/13      openfoam/v2312  &lt;br /&gt;
openfoam/10  openfoam/13-amd  openfoam/v2406  &lt;br /&gt;
openfoam/11  openfoam/v2212   &lt;br /&gt;
openfoam/12  openfoam/v2306&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
Contact us if you encounter problems; there can be various reasons why OpenFOAM may have trouble on certain hardware or when compiling dynamic code. When loading OpenFOAM modules, a number of dependencies will be automatically loaded for you, and you don&#039;t have to load those yourself. For example:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
$ module load openfoam/13&lt;br /&gt;
Loading openfoam/13&lt;br /&gt;
  Loading requirement: intel/2024.2.0/mpi/2021.13 gcc/gcc-12.1.0&lt;br /&gt;
&lt;br /&gt;
$ module list&lt;br /&gt;
Currently Loaded Modulefiles:&lt;br /&gt;
 1) intel/2024.2.0/mpi/2021.13   2) gcc/gcc-12.1.0   3) openfoam/13&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
In this case, OpenFOAM 13 loads the Intel 2024 MPI module, and loads the GCC compiler 12.1. OpenFOAM was compiled from source, and has been compiled specifically with that compiler and MPI version, so it make little sense to use other compilers or MPI libraries.&lt;br /&gt;
&lt;br /&gt;
Note: We have found a problem with running the Intel 2024 MPI library in the amd64 queue. Therefore, we have a modified module that uses the Intel 2022 library (I know -- Intel 2022 gives you the 2021 MPI libraries, but that is the way Intel distributes this software):&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
$ module load openfoam/13-amd &lt;br /&gt;
Loading mpi version 2021.7.0&lt;br /&gt;
Loading openfoam/13-amd&lt;br /&gt;
  Loading requirement: intel/2022.2.0/mpi/2021.7.0 gcc/gcc-12.1.0&lt;br /&gt;
&lt;br /&gt;
$ module list&lt;br /&gt;
Currently Loaded Modulefiles:&lt;br /&gt;
 1) intel/2022.2.0/mpi/2021.7.0   2) gcc/gcc-12.1.0   3) openfoam/13-amd&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
If you are compiling OpenFOAM yourself, the modules are of little help. You would need to select the appropriate MPI version and compiler before doing so, and then consistently load them before running your OpenFOAM executables. Within the &amp;quot;etc/bashrc&amp;quot; file in the source code tree, you want to set the MPI library to INTELMPI. As usual with self-compiled versions of OpenFOAM, you would &amp;quot;source etc/bashrc&amp;quot; to set up your personal environment to run your home-brew version of OpenFOAM. Contact us if you need to learn more about compiling OpenFOAM on the system.&lt;br /&gt;
&lt;br /&gt;
==Additional Software Applications and Libraries==&lt;br /&gt;
&lt;br /&gt;
===Loadable GCC Compiler Versions===&lt;br /&gt;
&lt;br /&gt;
The Rocky 9.6 operating system uses the GCC 11.5 compiler. That should be sufficient for most users when compiling your own applications. In case you need to use either a more up-to-date compiler, or if you need an older compiler for compatibility, we make the following versions available as loadable modules.&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
$ module avail gcc&lt;br /&gt;
------------ /shared/apps/modulefiles ------------&lt;br /&gt;
gcc/gcc-4.9.4  gcc/gcc-7.5.0  gcc/gcc-10.3.0  &lt;br /&gt;
gcc/gcc-5.5.0  gcc/gcc-8.5.0  gcc/gcc-11.3.0  &lt;br /&gt;
gcc/gcc-6.5.0  gcc/gcc-9.5.0  gcc/gcc-12.1.0&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
Additional versions can be created and made available as modules as well. If you need a specific version that is not currently available, please ask us to compiler and install it. If necessary, we may be able to provide access to other compilers, for example LLVM. We do not provide access to proprietary compilers at this time.&lt;br /&gt;
&lt;br /&gt;
===MPI Libraries and Runtimes===&lt;br /&gt;
&lt;br /&gt;
While we seem to have a variety of MPI versions and flavors available to users, the only MPI versions that allow us to run software over Infiniband are the Intel MPI libraries. Some of the installed alternatives are likely to fail, or will have a set of environment variables that have to be set. All major engineering software packages that we offer are pre-configured with specific MPI versions and settings that have been tested and/or provided by the vendors.&lt;br /&gt;
&lt;br /&gt;
Note: Some MPI libraries may seem to work. They may allow your MPI application to run. But inter-process network communication may travel through the rather slow and high-latency Ethernet fabric, making MPI applications very ineffective and are probably not worth while.&lt;br /&gt;
&lt;br /&gt;
===MatLab Runtimes===&lt;br /&gt;
&lt;br /&gt;
We can install MatLAB run time libraries as needed and have them available as loadable modules. Recently, we had a problem with MatLAB run time libraries being identified as security vulnerabilities. Contact us if you need them installed for one of your projects.&lt;br /&gt;
&lt;br /&gt;
===Anaconda and variants (miniconda etc)===&lt;br /&gt;
&lt;br /&gt;
Our current practice is to have users download and install their own versions of Anaconda and its variants in their own home directories. This allows for maximum flexibility when it comes to installable software modules, and users can maintain the installation, upgrades, and maintenance themselves. If you encounter issues, please contact us. One known side effect of Anaconda installations is a performance hit when starting your software, e.g. python scripts may take 30 seconds or more to execute. This is an artefact caused by the Lustre file system, which has been designed for large files accessible from many machines simultaneously. Performance on reading many small files has not been considered and is fairly poor. Again, contact us and we will design a solution for you as needed.&lt;/div&gt;</summary>
		<author><name>Ley</name></author>
	</entry>
	<entry>
		<id>https://wiki.anl.gov/wiki_tracc/index.php?title=Job_Submission_and_Monitoring&amp;diff=2486</id>
		<title>Job Submission and Monitoring</title>
		<link rel="alternate" type="text/html" href="https://wiki.anl.gov/wiki_tracc/index.php?title=Job_Submission_and_Monitoring&amp;diff=2486"/>
		<updated>2026-02-21T19:17:25Z</updated>

		<summary type="html">&lt;p&gt;Ley: /* Submitting an LS-Dyna Job */&lt;/p&gt;
&lt;hr /&gt;
&lt;div&gt;{| align=&amp;quot;right&amp;quot;&lt;br /&gt;
| __TOC__&lt;br /&gt;
|}&lt;br /&gt;
==Resource Summary View==&lt;br /&gt;
&lt;br /&gt;
To get started, users can query the overall status of resources on the cluster. The &#039;&#039;&#039;&amp;quot;qsum&amp;quot;&#039;&#039;&#039; script will list all queues and nodes, as well as how many are offline, down, free, or assigned to users. This is a script developed by our team, and may need to be updated if something goes wrong. Please contact us if you experience any problems.&lt;br /&gt;
&lt;br /&gt;
Each queue groups a number of nodes together based on their hardware and software configurations. Nodes can be part of more than one queue, and there are other complex details that we are ignoring here for the purpose of keeping it simple.&lt;br /&gt;
&lt;br /&gt;
===Queues===&lt;br /&gt;
&lt;br /&gt;
Here is a very brief summary of what each of the queues is, and how to use them efficiently:&lt;br /&gt;
&lt;br /&gt;
; a4000: This is a queue that has three 16-core CPU machines, each of which is furthermore equipped with three A4000 GPUs. That makes a total of 9 A4000 GPUs available to users. Neither the GPUs nor the processors are particularly powerful these days. The machines have 500GB of memory though, which makes for a good platform for experimenting with GPU capabilities.&lt;br /&gt;
; a6000: This is a queue that has only one 64-core CPU machines, and is equipped with four A6000 GPUs. The system can be upgraded to 8 A6000 GPUs if needed. This is a decent GPU machine that can take a solid workload these days. The machine has 750GB of memory, which makes for a good production platform.&lt;br /&gt;
; amd16: This is a queue with many of our older AMD-based 16-core machines, each of which has 30GB of memory. While individual machines are a bit outdated, they are all interconnected with Infiniband and can provide a solid production workload in multi-nodes jobs over MPI without blocking the more current (and thus expensive) systems.&lt;br /&gt;
; epyc1/epyc2: These are 2 separate queues with slightly different performance characteristics. Each of the groups is interconnected with Infiniband to provide a platform for large and demanding software packages, such as LS-Dyna and StarCCM+. They have between 250GB and 500GB of memory. Because licenses for these software packages are very expensive, they should use these two queues for making optimum use of limited core licenses available to each package.&lt;br /&gt;
; xeon28: This is a set of intermediate machines with 28 cores and 64GB of memory. They can be used for a variety of purposes, including MPI jobs and single node application software.&lt;br /&gt;
; virtual: This is a set of nodes without MPI capabilities. They are virtual machines with 32GB each. They can be used for higher demand applications that would interfere with the login nodes, and therefore with other users of these login machines. A user would submit interactive jobs to individual virtual machines and avoid any significant load on login nodes.&lt;br /&gt;
&lt;br /&gt;
===The Queue Summary Script (qsum)===&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
$ qsum&lt;br /&gt;
=============== a4000 ==========================================================&lt;br /&gt;
Queue: &amp;quot;a4000&amp;quot; / nodes: 3 / down: 0 / offline: 0 / busy: 0 / available: 3&lt;br /&gt;
      AVAILABLE (3): g001, g002, g003&lt;br /&gt;
=============== a6000 ==========================================================&lt;br /&gt;
Queue: &amp;quot;a6000&amp;quot; / nodes: 1 / down: 0 / offline: 0 / busy: 0 / available: 1&lt;br /&gt;
      AVAILABLE (1): lambda01&lt;br /&gt;
=============== amd16 ==========================================================&lt;br /&gt;
Queue: &amp;quot;amd16&amp;quot; / nodes: 33 / down: 2 / offline: 0 / busy: 2 / available: 29&lt;br /&gt;
           DOWN (2): n017, n030&lt;br /&gt;
            ley (2): n001, n002&lt;br /&gt;
     AVAILABLE (29): n003, n004, n005, n006, n007, n008, n009, n010, n011, n012&lt;br /&gt;
                     n013, n014, n015, n016, n018, n019, n020, n021, n022, n023&lt;br /&gt;
                     n024, n025, n026, n027, n028, n029, n031, n032, n039&lt;br /&gt;
=============== epyc1 ==========================================================&lt;br /&gt;
Queue: &amp;quot;epyc1&amp;quot; / nodes: 1 / down: 0 / offline: 0 / busy: 0 / available: 1&lt;br /&gt;
      AVAILABLE (1): a027&lt;br /&gt;
=============== epyc2 ==========================================================&lt;br /&gt;
Queue: &amp;quot;epyc2&amp;quot; / nodes: 20 / down: 0 / offline: 0 / busy: 5 / available: 15&lt;br /&gt;
            ley (2): a030, a031&lt;br /&gt;
         msitek (3): a028, a029, a032&lt;br /&gt;
     AVAILABLE (15): a033, a034, a035, a036, a037, a038, a039, a040, a041, a042&lt;br /&gt;
                     a043, a044, a045, a046, a047&lt;br /&gt;
=============== virtual ========================================================&lt;br /&gt;
Queue: &amp;quot;virtual&amp;quot; / nodes: 6 / down: 0 / offline: 0 / busy: 0 / available: 6&lt;br /&gt;
      AVAILABLE (6): v001, v002, v003, v004, v005, v006&lt;br /&gt;
=============== xeon28 =========================================================&lt;br /&gt;
Queue: &amp;quot;xeon28&amp;quot; / nodes: 12 / down: 0 / offline: 0 / busy: 0 / available: 12&lt;br /&gt;
     AVAILABLE (12): p001, p002, p003, p004, p005, p006, p007, p008, p009, p010&lt;br /&gt;
                     p011, p012&lt;br /&gt;
================================================================================&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
==Queue Status and Monitoring Jobs==&lt;br /&gt;
&lt;br /&gt;
===qstat===&lt;br /&gt;
&lt;br /&gt;
To find out out about the status of all running jobs on the cluster you can use the &#039;&#039;&#039;&amp;quot;qstat&amp;quot;&#039;&#039;&#039; command. Here is an example:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
$ qstat&lt;br /&gt;
&lt;br /&gt;
Nov 20 18:30 ley@login3:Plots$ qstat&lt;br /&gt;
Job id            Name             User              Time Use S Queue&lt;br /&gt;
----------------  ---------------- ----------------  -------- - -----&lt;br /&gt;
3023.pbs          STDIN            msitek            4144:14* R epyc2           &lt;br /&gt;
3029.pbs          STDIN            ley               76:46:53 R epyc2           &lt;br /&gt;
3032.pbs          STDIN            msitek            2879:52* R epyc2           &lt;br /&gt;
3033.pbs          STDIN            msitek            3687:29* R epyc2           &lt;br /&gt;
3048.pbs          foo.sh           james.cook               0 Q amd16           &lt;br /&gt;
3060.pbs          of13.sh          ley               310:47:* R epyc2           &lt;br /&gt;
3061.pbs          of13.sh          ley               308:37:* R epyc2           &lt;br /&gt;
3062.pbs          of13.sh          ley               308:02:* R epyc2           &lt;br /&gt;
3063.pbs          of13.sh          ley               308:15:* R epyc2&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
The first column shows the &#039;&#039;&#039;job id&#039;&#039;&#039;, a unique identifier for all jobs ever submitted to the cluster. This job id is important when killing jobs, or for other actions you may need to take.&lt;br /&gt;
&lt;br /&gt;
The next column shows the name of the job script. If the column shows &#039;&#039;&#039;STDIN&#039;&#039;&#039;, it means that this is an &#039;&#039;&#039;interactive job&#039;&#039;&#039; where a user can enter commands in a terminal window. This is particularly useful for model and software development task where the application has to be started and killed repeatedly.&lt;br /&gt;
&lt;br /&gt;
The owner of the job is shown next. These are the user names of the various people using the cluster.&lt;br /&gt;
&lt;br /&gt;
The last three columns indicate the current run time of the job, whether it is running (R) or waiting (Q) for execution. The last entry shows the queue in which the job is running.&lt;br /&gt;
&lt;br /&gt;
===qstat -an1===&lt;br /&gt;
&lt;br /&gt;
Adding a few options gives much more detail about each jobs.&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
qstat -an1&lt;br /&gt;
&lt;br /&gt;
Nov 20 13:09 ley@login3:Plots$ qstat -an1&lt;br /&gt;
&lt;br /&gt;
pbs: &lt;br /&gt;
                                                            Req&#039;d  Req&#039;d   Elap&lt;br /&gt;
Job ID          Username Queue    Jobname    SessID NDS TSK Memory Time  S Time&lt;br /&gt;
--------------- -------- -------- ---------- ------ --- --- ------ ----- - -----&lt;br /&gt;
3023.pbs        msitek   epyc2    STDIN      24360*   1  64  350gb 100:0 R 81:46 a028/0*64&lt;br /&gt;
3029.pbs        ley      epyc2    STDIN      21719*   2 128  100gb 200:0 R 72:31 a030/0*64+a031/0*64&lt;br /&gt;
3032.pbs        msitek   epyc2    STDIN      18102*   1  64  350gb 100:0 R 57:57 a029/0*64&lt;br /&gt;
3033.pbs        msitek   epyc2    STDIN      830486   1  64  350gb 100:0 R 57:53 a032/0*64&lt;br /&gt;
3048.pbs        james.c* amd16    foo.sh        --    1  28   30gb 01:00 Q   --   -- &lt;br /&gt;
3060.pbs        ley      epyc2    STDIN      763101   1  64  350gb 48:00 R 06:42 a033/0*64&lt;br /&gt;
3061.pbs        ley      epyc2    STDIN      763947   1  64  350gb 48:00 R 06:40 a034/0*64&lt;br /&gt;
3062.pbs        ley      epyc2    STDIN      761473   1  64  350gb 48:00 R 06:39 a035/0*64&lt;br /&gt;
3063.pbs        ley      epyc2    STDIN      766205   1  64  350gb 48:00 R 06:40 a036/0*64&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
In this table, you can see how many nodes and cores are being used by each job. For example, job 3029 of the user &amp;quot;ley&amp;quot; shows that it is running on 2 nodes using a total of 128 cores. In addition to the elapsed time, the table also show the reserved time for this job. This allow you to estimate when a job will be definitely finalized (or killed by the system if still running).&lt;br /&gt;
&lt;br /&gt;
The last column (without a header) is written because the option &#039;&#039;&#039;&amp;quot;-an1&amp;quot;&#039;&#039;&#039; was used. This useful to learn about which nodes are used by each job.&lt;br /&gt;
&lt;br /&gt;
===qstat -q===&lt;br /&gt;
&lt;br /&gt;
To learn more about the queues on the cluster, the &#039;&#039;&#039;&amp;quot;-q&amp;quot;&#039;&#039;&#039; option turns out to be useful.&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
$ qstat -q&lt;br /&gt;
&lt;br /&gt;
server: pbs&lt;br /&gt;
&lt;br /&gt;
Queue            Memory CPU Time Walltime Node   Run   Que   Lm  State&lt;br /&gt;
---------------- ------ -------- -------- ---- ----- ----- ----  -----&lt;br /&gt;
virtual            30gb    --       --       1     0     0   --   E R&lt;br /&gt;
a4000             500gb    --       --       1     0     0   --   E R&lt;br /&gt;
a6000             750gb    --       --       1     0     0   --   E R&lt;br /&gt;
xeon28             --      --       --       4     0     0   --   E R&lt;br /&gt;
amd16              --      --       --       8     0     1   --   E R&lt;br /&gt;
epyc2              --      --       --       2    14     0   --   E R&lt;br /&gt;
epyc1              --      --       --       2     0     0   --   E R&lt;br /&gt;
                                               ----- -----&lt;br /&gt;
                                                  14     1&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
For each queue, some basic values are displayed. The first three queues listed above have a default memory allocation as shown, and the column &#039;&#039;&#039;&amp;quot;Node&amp;quot;&#039;&#039;&#039; indicates the maximum number of nodes that can be asked for at job submission time. For example, you can request just one node for a job from the first three queues (because these are queues where MPI makes no sense). The &#039;&#039;&#039;&amp;quot;xeon28&amp;quot;&#039;&#039;&#039; queue also for a maximum of 4 nodes per MPI job. The &#039;&#039;&#039;&amp;quot;amd16&amp;quot;&#039;&#039;&#039; queue has a maximum of 8 nodes per job, and the &#039;&#039;&#039;&amp;quot;epyc1&amp;quot;&#039;&#039;&#039; and &#039;&#039;&#039;&amp;quot;epyc2&amp;quot;&#039;&#039;&#039; queues have maxima of two nodes per job. These limitations can be changed by the administrator as needed. As shown above, this will prevent inefficient resource requests.&lt;br /&gt;
&lt;br /&gt;
===qstat -f===&lt;br /&gt;
&lt;br /&gt;
The command &#039;&#039;&#039;&amp;quot;qstat -f -F json 3029&amp;quot;&#039;&#039;&#039; retrieves extremely detailed stats on the running job 3029. The result can be returned in JSON format to be ready for further processing (shown below).&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
$ qstat -f -F json 3029&lt;br /&gt;
{&lt;br /&gt;
    &amp;quot;timestamp&amp;quot;:1763705353,&lt;br /&gt;
    &amp;quot;pbs_version&amp;quot;:&amp;quot;23.06.06&amp;quot;,&lt;br /&gt;
    &amp;quot;pbs_server&amp;quot;:&amp;quot;pbs&amp;quot;,&lt;br /&gt;
    &amp;quot;Jobs&amp;quot;:{&lt;br /&gt;
        &amp;quot;3029.pbs&amp;quot;:{&lt;br /&gt;
            &amp;quot;Job_Name&amp;quot;:&amp;quot;STDIN&amp;quot;,&lt;br /&gt;
            &amp;quot;Job_Owner&amp;quot;:&amp;quot;ley@login4&amp;quot;,&lt;br /&gt;
            &amp;quot;resources_used&amp;quot;:{&lt;br /&gt;
                &amp;quot;cpupercent&amp;quot;:98,&lt;br /&gt;
                &amp;quot;cput&amp;quot;:&amp;quot;76:46:53&amp;quot;,&lt;br /&gt;
                &amp;quot;hpmem&amp;quot;:&amp;quot;0b&amp;quot;,&lt;br /&gt;
                &amp;quot;mem&amp;quot;:&amp;quot;52428800kb&amp;quot;,&lt;br /&gt;
                &amp;quot;ncpus&amp;quot;:128,&lt;br /&gt;
                &amp;quot;vmem&amp;quot;:&amp;quot;52428800kb&amp;quot;,&lt;br /&gt;
                &amp;quot;walltime&amp;quot;:&amp;quot;78:09:32&amp;quot;&lt;br /&gt;
            },&lt;br /&gt;
            &amp;quot;job_state&amp;quot;:&amp;quot;R&amp;quot;,&lt;br /&gt;
            &amp;quot;queue&amp;quot;:&amp;quot;epyc2&amp;quot;,&lt;br /&gt;
            &amp;quot;server&amp;quot;:&amp;quot;pbs&amp;quot;,&lt;br /&gt;
            &amp;quot;Checkpoint&amp;quot;:&amp;quot;u&amp;quot;,&lt;br /&gt;
            &amp;quot;ctime&amp;quot;:&amp;quot;Mon Nov 17 17:58:25 2025&amp;quot;,&lt;br /&gt;
            &amp;quot;Error_Path&amp;quot;:&amp;quot;/dev/pts/0&amp;quot;,&lt;br /&gt;
            &amp;quot;exec_host&amp;quot;:&amp;quot;a030/0*64+a031/0*64&amp;quot;,&lt;br /&gt;
            &amp;quot;exec_vnode&amp;quot;:&amp;quot;(a030:ncpus=64:mem=52428800kb)+(a031:ncpus=64:mem=52428800kb)&amp;quot;,&lt;br /&gt;
            &amp;quot;Hold_Types&amp;quot;:&amp;quot;n&amp;quot;,&lt;br /&gt;
            &amp;quot;interactive&amp;quot;:&amp;quot;True&amp;quot;,&lt;br /&gt;
            &amp;quot;Join_Path&amp;quot;:&amp;quot;n&amp;quot;,&lt;br /&gt;
            &amp;quot;Keep_Files&amp;quot;:&amp;quot;n&amp;quot;,&lt;br /&gt;
            &amp;quot;Mail_Points&amp;quot;:&amp;quot;a&amp;quot;,&lt;br /&gt;
            &amp;quot;mtime&amp;quot;:&amp;quot;Fri Nov 21 00:07:59 2025&amp;quot;,&lt;br /&gt;
            &amp;quot;Output_Path&amp;quot;:&amp;quot;/dev/pts/0&amp;quot;,&lt;br /&gt;
            &amp;quot;Priority&amp;quot;:0,&lt;br /&gt;
            &amp;quot;qtime&amp;quot;:&amp;quot;Mon Nov 17 17:58:25 2025&amp;quot;,&lt;br /&gt;
            &amp;quot;Rerunable&amp;quot;:&amp;quot;False&amp;quot;,&lt;br /&gt;
            &amp;quot;Resource_List&amp;quot;:{&lt;br /&gt;
                &amp;quot;mem&amp;quot;:&amp;quot;100gb&amp;quot;,&lt;br /&gt;
                &amp;quot;mpiprocs&amp;quot;:128,&lt;br /&gt;
                &amp;quot;ncpus&amp;quot;:128,&lt;br /&gt;
                &amp;quot;nodect&amp;quot;:2,&lt;br /&gt;
                &amp;quot;place&amp;quot;:&amp;quot;free&amp;quot;,&lt;br /&gt;
                &amp;quot;select&amp;quot;:&amp;quot;2:ncpus=64:mem=50gb:mpiprocs=64&amp;quot;,&lt;br /&gt;
                &amp;quot;walltime&amp;quot;:&amp;quot;200:00:00&amp;quot;&lt;br /&gt;
            },&lt;br /&gt;
            &amp;quot;stime&amp;quot;:&amp;quot;Mon Nov 17 17:58:25 2025&amp;quot;,&lt;br /&gt;
            &amp;quot;session_id&amp;quot;:2171964,&lt;br /&gt;
            &amp;quot;jobdir&amp;quot;:&amp;quot;/mnt/lustre/arrow/home/ley&amp;quot;,&lt;br /&gt;
            &amp;quot;substate&amp;quot;:42,&lt;br /&gt;
            &amp;quot;Variable_List&amp;quot;:{&lt;br /&gt;
                &amp;quot;PBS_O_HOME&amp;quot;:&amp;quot;/mnt/lustre/arrow/home/ley&amp;quot;,&lt;br /&gt;
                &amp;quot;PBS_O_LANG&amp;quot;:&amp;quot;en_US.UTF-8&amp;quot;,&lt;br /&gt;
                &amp;quot;PBS_O_LOGNAME&amp;quot;:&amp;quot;ley&amp;quot;,&lt;br /&gt;
                &amp;quot;PBS_O_PATH&amp;quot;:&amp;quot;/shared/apps/active/lstc/lsprepost/SP-4.5:/shared/apps/active/lstc/lsprepost/DP-4.3:/shared/bin:/usr/share/Modules/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/pbs/bin:/opt/thinlinc/bin:/opt/thinlinc/sbin:/mnt/lustre/arrow/home/ley/.local/bin:/mnt/lustre/arrow/home/ley/bin&amp;quot;,&lt;br /&gt;
                &amp;quot;PBS_O_MAIL&amp;quot;:&amp;quot;/var/spool/mail/ley&amp;quot;,&lt;br /&gt;
                &amp;quot;PBS_O_SHELL&amp;quot;:&amp;quot;/bin/bash&amp;quot;,&lt;br /&gt;
                &amp;quot;PBS_O_WORKDIR&amp;quot;:&amp;quot;/mnt/lustre/arrow/home/ley/Qualification/LS-Dyna/Rocky9/Seatbelt/Template&amp;quot;,&lt;br /&gt;
                &amp;quot;PBS_O_SYSTEM&amp;quot;:&amp;quot;Linux&amp;quot;,&lt;br /&gt;
                &amp;quot;PBS_O_QUEUE&amp;quot;:&amp;quot;epyc2&amp;quot;,&lt;br /&gt;
                &amp;quot;PBS_O_HOST&amp;quot;:&amp;quot;login4&amp;quot;&lt;br /&gt;
            },&lt;br /&gt;
            &amp;quot;comment&amp;quot;:&amp;quot;Job run at Mon Nov 17 at 17:58 on (a030:ncpus=64:mem=52428800kb)+(a031:ncpus=64:mem=52428800kb)&amp;quot;,&lt;br /&gt;
            &amp;quot;etime&amp;quot;:&amp;quot;Mon Nov 17 17:58:25 2025&amp;quot;,&lt;br /&gt;
            &amp;quot;run_count&amp;quot;:1,&lt;br /&gt;
            &amp;quot;Submit_arguments&amp;quot;:&amp;quot;-I -q epyc2 -l walltime=200:00:00,select=2:ncpus=64:mem=50gb:mpiprocs=64&amp;quot;,&lt;br /&gt;
            &amp;quot;project&amp;quot;:&amp;quot;_pbs_project_default&amp;quot;,&lt;br /&gt;
            &amp;quot;Submit_Host&amp;quot;:&amp;quot;login4&amp;quot;&lt;br /&gt;
        }&lt;br /&gt;
    }&lt;br /&gt;
}&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
===Manual pages for qstat===&lt;br /&gt;
&lt;br /&gt;
To learn more about the &#039;&#039;&#039;&amp;quot;qstat&amp;quot;&#039;&#039;&#039; command, you can use the command &#039;&#039;&#039;&amp;quot;man qstat&amp;quot;&#039;&#039;&#039;, which will print a lot of detailed information about the capabilities of this command.&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
$ man qstat&lt;br /&gt;
&lt;br /&gt;
qstat(1B)                                       PBS Professional                                       qstat(1B)&lt;br /&gt;
&lt;br /&gt;
NAME&lt;br /&gt;
       qstat - display status of PBS jobs, queues, or servers&lt;br /&gt;
&lt;br /&gt;
SYNOPSIS&lt;br /&gt;
       Displaying Job Status&lt;br /&gt;
              Default format:&lt;br /&gt;
              qstat [-E] [-J] [-p] [-t] [-w] [-x] [[&amp;lt;job ID&amp;gt; | &amp;lt;destination&amp;gt;] ...]&lt;br /&gt;
&lt;br /&gt;
              Long format:&lt;br /&gt;
              qstat -f [-F json | dsv [-D &amp;lt;delimiter&amp;gt;]] [-E] [-J] [-p] [-t] [-w]&lt;br /&gt;
                    [-x] [[&amp;lt;job ID&amp;gt; | &amp;lt;destination&amp;gt;] ...]&lt;br /&gt;
... &amp;lt;many more pages&amp;gt;&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
==Job Submission Basics==&lt;br /&gt;
&lt;br /&gt;
Jobs are submitted into the system using the &#039;&#039;&#039;&amp;quot;qsub&amp;quot;&#039;&#039;&#039; application. This application can take many different options and allows for a lot of different resource requests to tell the cluster what to do. We are running &#039;&#039;&#039;OpenPBS 23.06.06&#039;&#039;&#039; as our job scheduler. Here is a link to the User&#039;s Manual (of PBS PRO) if you want to explore gory details and capabilities. The User&#039;s Guide has about 240 pages, the Reference Guide has 500 pages, and the Big Book has 2500 pages. So there is a lot of information available. I also added job submission info for the LCRC cluster.&lt;br /&gt;
&lt;br /&gt;
* [https://argonne-lcrc.github.io/user-guides/running-jobs-at-lcrc/pbs-pro/ Argonne&#039;s LCRC pages on job submissions on their clusters]&lt;br /&gt;
* [https://help.altair.com/2022.1.0/PBS%20Professional/PBSUserGuide2022.1.pdf PBS Professional 2022.1 User&#039;s Guide]&lt;br /&gt;
* [https://help.altair.com/2022.1.0/PBS%20Professional/PBSReferenceGuide2022.1.pdf PBS Professional 2022.1 Reference Guide]&lt;br /&gt;
* [https://help.altair.com/2022.1.0/PBS%20Professional/PBS2022.1.pdf Altair PBS Professional 2022.1 Big Book]&lt;br /&gt;
&lt;br /&gt;
The User&#039;s Guide can be very helpful to clarify some of the concepts and capabilities, but it can be hard to find the specific information you may be looking for. Please understand that we are no longer running TORQUE and MAUI, so the syntax for job submission is distinctively different yet quite similar.&lt;br /&gt;
&lt;br /&gt;
The reference guide may be helpful to understand the complete syntax and full capabilities of the software.&lt;br /&gt;
&lt;br /&gt;
The big book is what I had to use when configuring OpenPBS earlier this year. This includes all the tricky details needed to make the system work smoothly for us. It&#039;s a bit scary to look at a PDF file that is 2500 pages long, but that is nothing compared to the StarCCM+ manuals.&lt;br /&gt;
&lt;br /&gt;
&amp;lt;BLOCKQUOTE&amp;gt;&lt;br /&gt;
[[File:Attention.jpg|25px]] &#039;&#039;&#039;IMPORTANT NOTE:&#039;&#039;&#039; &#039;&#039;The following sections are important to understand. They explain how jobs are submitted and then scjeduled for execution based on resources available and the specific need of the user.&#039;&#039;&lt;br /&gt;
&amp;lt;/BLOCKQUOTE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
The following sections explain the various tasks you may want to submit fir execution, ordered from simple to complex.&lt;br /&gt;
&lt;br /&gt;
* General Batch Jobs&lt;br /&gt;
** Requesting a Single Node for a Job&lt;br /&gt;
** Requesting Multiple Nodes for a Job&lt;br /&gt;
* Embedded Job Resource Requests&lt;br /&gt;
* Interactive Jobs&lt;br /&gt;
* Interactive Jobs with X-Windows GUI Applications&lt;br /&gt;
* Running Multiple Jobs on Single Nodes&lt;br /&gt;
* Running Jobs using GPUs&lt;br /&gt;
&lt;br /&gt;
===General Batch Jobs===&lt;br /&gt;
&lt;br /&gt;
Let&#039;s get started with a very basic usage of the system. Let&#039;s assume you have a simple application, and you want to execute it on a cluster node. Let&#039;s also assume that this is a very simple application, one that runs on one or a few cores, doesn&#039;t require any keyboard interaction with the user, doesn&#039;t need the user to see what&#039;s typically written to the screen, and writes its output to files. In this case, we can submit this application as a batch job, which will place it into an execution queue and process it as soon as a node becomes available.&lt;br /&gt;
&lt;br /&gt;
If the application requires more cores than a single node can provide, we can run the application over Infiniband with MPI message passing. In this case, we need to understand the concept of MPI applications a bit better. In both cases, we get started by creating a folder on the file system. Naming conventions are important so that you can distinguish the jobs by folder name.&lt;br /&gt;
&lt;br /&gt;
For both of the above scenarios, you would typically create a Bash shell script, and then submit the script into one of the queues for eventual execution.&lt;br /&gt;
&lt;br /&gt;
====Requesting a Single Node for a Job====&lt;br /&gt;
&lt;br /&gt;
Let&#039;s try something rather trivial to get used to the concept. Create yourself a folder, for example &#039;&#039;&#039;&amp;quot;myjobfolder&amp;quot;&#039;&#039;&#039;. Within that folder, create a job submission script. That script can have any name, but something short and simple may be best. Let&#039;s assume you create a file called &#039;&#039;&#039;&amp;quot;cluster.job&amp;quot;&#039;&#039;&#039;. The file doesn&#039;t have to have that extension. Any file name will do. But using the same filename for all of your jobs helps finding your way around the many files that will be created over time. The &#039;&#039;&#039;&amp;quot;cluster.job&amp;quot;&#039;&#039;&#039; file should look something like this:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
#!/bin/bash&lt;br /&gt;
#&lt;br /&gt;
# the following ensures that you will change into the directory where you are&lt;br /&gt;
# submitting the job from.&lt;br /&gt;
cd $PBS_O_WORKDIR&lt;br /&gt;
#&lt;br /&gt;
# now we sleep for 60 seconds and waste time. This is a placeholder for your application,&lt;br /&gt;
# which would be doing useful work for you.&lt;br /&gt;
sleep 60&lt;br /&gt;
#&lt;br /&gt;
# and after doing things, we may want to write something into a file to show that&lt;br /&gt;
# our jobs is done.&lt;br /&gt;
echo `date` &amp;gt; info.log&lt;br /&gt;
#&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
This can be submitted without detailed resource specifications (except for the walltime, which is be default 0:00:00):&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
$ qsub -q virtual -l walltime=1:00:00 cluster.job &lt;br /&gt;
3072.pbs&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
Wait a little, then check the status of running jobs:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
$ qstat -an1&lt;br /&gt;
&lt;br /&gt;
pbs: &lt;br /&gt;
                                                            Req&#039;d  Req&#039;d   Elap&lt;br /&gt;
Job ID          Username Queue    Jobname    SessID NDS TSK Memory Time  S Time&lt;br /&gt;
--------------- -------- -------- ---------- ------ --- --- ------ ----- - -----&lt;br /&gt;
3023.pbs        msitek   epyc2    STDIN      24360*   1  64  350gb 100:0 R 83:17 a028/0*64&lt;br /&gt;
3029.pbs        ley      epyc2    STDIN      21719*   2 128  100gb 200:0 R 74:00 a030/0*64+a031/0*64&lt;br /&gt;
3033.pbs        msitek   epyc2    STDIN      830486   1  64  350gb 100:0 R 59:23 a032/0*64&lt;br /&gt;
3048.pbs        james.c* amd16    foo.sh        --    1  28   30gb 01:00 Q   --   -- &lt;br /&gt;
3060.pbs        ley      epyc2    STDIN      763101   1  64  350gb 48:00 R 08:10 a033/0*64&lt;br /&gt;
3061.pbs        ley      epyc2    STDIN      763947   1  64  350gb 48:00 R 08:10 a034/0*64&lt;br /&gt;
3070.pbs        ley      epyc2    STDIN      766847   1  64  350gb 48:00 R 07:23 a042/0*64&lt;br /&gt;
3072.pbs        ley      virtual  cluster.j* 230230   1   4   30gb 01:00 E 00:01 v001/0*4&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
In this particular example, we are sending this job to the &#039;&#039;&#039;queue &amp;quot;virtual&amp;quot;&#039;&#039;&#039;. This queue, by default, allocates 30GB of memory to the job, and runs on 1 node with 4 cores. This is sufficient capacity to run quite a workload. When submitting a job to a single node, &#039;&#039;&#039;reasonable maximum allocations are automatically assigned&#039;&#039;&#039;, and the user doesn&#039;t have to worry about running out of memory or how many cores he will be using.&lt;br /&gt;
&lt;br /&gt;
The only required argument is the &#039;&#039;&#039;&amp;quot;walltime&amp;quot;&#039;&#039;&#039; argument. By default, the job will quit as soon as it is submitted. This indicates to the user that he forgot to provide the &#039;&#039;&#039;&amp;quot;walltime&amp;quot;&#039;&#039;&#039; argument.&lt;br /&gt;
&lt;br /&gt;
When the job disappears from the job list, it is done. At this point, you will find the file &amp;quot;info.log&amp;quot; in your job folder.&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
$ cat info.log &lt;br /&gt;
Thu Nov 20 08:00:31 PM CST 2025&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
====Requesting Multiple Nodes for a Job====&lt;br /&gt;
&lt;br /&gt;
To run jobs on multiple nodes, you will be likely &#039;&#039;&#039;executing jobs using MPI&#039;&#039;&#039;, the message passing interface. This establishes high-speed low-latency interconnections between the cores on one machine and the cores on the other machines. Data transfer does not require involvement of the cores themselves. Instead, the core tell the InfiniBand interconnect (and cores on the same node through shared memory) to transfer the data through RDMA, remoted direct memory access. The cores don&#039;t need to spend CPU cycles on copying data, but rather simply access the data once it has been copied by the Infiniband fabric. This makes for extremely efficient remote memory access, and message passing is used to coordinate data transfer between the cores no matter where they are located on any of the nodes.&lt;br /&gt;
&lt;br /&gt;
On our cluster, MPI-aware applications like &#039;&#039;&#039;OpenFOAM&#039;&#039;&#039;, &#039;&#039;&#039;StarCCM+&#039;&#039;&#039;, and &#039;&#039;&#039;LS-Dyna&#039;&#039;&#039; can be loaded as modules, which then automatically selects the most appropriate MPI library to use. The software applications have been tested to ensure that they work out-of-the box if a user selects any specific version of any of the applications.&lt;br /&gt;
&lt;br /&gt;
The following is a very trivial example for the MPI execution of a very simple executable, with one copy running on each core of the nodes allocated to  the job. It doesn&#039;t perform any real work and just wastes resources for a short time, but it illustrates how execution on the cores of various nodes works.&lt;br /&gt;
&lt;br /&gt;
Like in the previous section, we start with a simple job script that we submit to an appropriate queues. In this case, we pick a queue that has machines with Infiniband interfaces supporting efficient communications. Let&#039;s assume we edit a file with the name &#039;&#039;&#039;&amp;quot;parallel.job&amp;quot;&#039;&#039;&#039; like this:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
#!/bin/bash&lt;br /&gt;
#&lt;br /&gt;
# the following ensures that you will change into the directory where you are&lt;br /&gt;
# submitting the job from.&lt;br /&gt;
cd $PBS_O_WORKDIR&lt;br /&gt;
#&lt;br /&gt;
# to execute a simple command on all of the cores of all of the nodes allocated to the job,&lt;br /&gt;
# we need to make one of the MPI versions available. Let&#039;s use one of the most up-to-date&lt;br /&gt;
# MPI library available on the cluster&lt;br /&gt;
module load intel/2024.2.0/mpi/2021.13&lt;br /&gt;
#&lt;br /&gt;
# now we are apply a few settings that ensure that the MPI library will use the highest-performing&lt;br /&gt;
# Infiniband Interconnect, as well as a few options to tell MPI how to interface nodes with&lt;br /&gt;
# each other and which specific Infiniband adapter to use. This is complex and requires in-depth&lt;br /&gt;
# knowledge of the QLogic Infiniband adapters we are using. It is unlikely that you will ever have to&lt;br /&gt;
# deal with these options, because the &amp;quot;module load&amp;quot; command for the engineering applications we provide&lt;br /&gt;
# on ARROW will handle all those details transparently without the user needing to understand the details.&lt;br /&gt;
export I_MPI_HYDRA_BOOTSTRAP=ssh&lt;br /&gt;
export MPI_DEVICE=rdma:ofa-v2-ib0&lt;br /&gt;
export UCX_NET_DEVICES=qib0:1&lt;br /&gt;
#&lt;br /&gt;
# it doesn&#039;t make much sense, but in this example we are executing the OS command &amp;quot;uptime&amp;quot; on all cores&lt;br /&gt;
# of the nodes allocated to this job. The output from each core is written to the file info.log. We&lt;br /&gt;
# will find 56 lines of output in the file info.log, each created by the corresponding core executing&lt;br /&gt;
# the uptime command.&lt;br /&gt;
mpirun uptime &amp;gt; info.log&lt;br /&gt;
#&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
A good queue to test scripts is the &#039;&#039;&#039;&amp;quot;xeon28&amp;quot;&#039;&#039;&#039; queue. In the queue, we have 2 14-core Xeon processers per node, so that means that each node has 56 actual cores. We do not consider hyperthreading when doing parallel computing. 56 actual cores is what&#039;s being used here. The job submission will look like this:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
qsub -q xeon28 -l walltime=1:0:0 -l select=2:ncpus=28:mpiprocs=28:mem=60G parallel.job&lt;br /&gt;
        ^                  ^               ^       ^           ^      ^ ^ ^&lt;br /&gt;
        |                  |               |       |           |      | | + --- the name of the job script to execute&lt;br /&gt;
        |                  |               |       |           |      | + ----- don&#039;t forget to specify gigabytes&lt;br /&gt;
        |                  |               |       |           |      + ------- the amount of memory to request per node&lt;br /&gt;
        |                  |               |       |           + -------------- the number of MPI tasks per nodes&lt;br /&gt;
        |                  |               |       + -------------------------- the number of cores per node&lt;br /&gt;
        |                  |               + ---------------------------------- the number of nodes to select in the queue&lt;br /&gt;
        |                   + ------------------------------------------------- the requested time, in this case 1h&lt;br /&gt;
        + --------------------------------------------------------------------- the queue to be used for the job&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
At this point, the job has created a file &amp;quot;info.log&amp;quot; with 56 lines, one per core:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
 22:26:05 up 23 days,  9:28,  0 users,  load average: 0.00, 0.00, 0.00&lt;br /&gt;
 22:26:05 up 23 days,  9:28,  0 users,  load average: 0.00, 0.00, 0.00&lt;br /&gt;
 22:26:05 up 23 days,  9:28,  0 users,  load average: 0.00, 0.00, 0.00&lt;br /&gt;
 22:26:05 up 23 days,  9:28,  0 users,  load average: 0.00, 0.00, 0.00&lt;br /&gt;
 22:26:05 up 23 days,  9:28,  0 users,  load average: 0.00, 0.00, 0.00&lt;br /&gt;
 22:26:05 up 23 days,  9:28,  0 users,  load average: 0.00, 0.00, 0.00&lt;br /&gt;
 22:26:05 up 23 days,  9:53,  0 users,  load average: 0.06, 0.03, 0.00&lt;br /&gt;
 22:26:05 up 23 days,  9:53,  0 users,  load average: 0.06, 0.03, 0.00&lt;br /&gt;
 22:26:05 up 23 days,  9:53,  0 users,  load average: 0.06, 0.03, 0.00&lt;br /&gt;
...&lt;br /&gt;
&amp;lt;/PRE&amp;gt; &lt;br /&gt;
&lt;br /&gt;
In this simple example, the lines look all the same. Upon close examination through, you can find slightly different values for some of the lines. Some lines say that the machine is up for 23 days and 9:28, while others say 23 days and 9:53. Because all 28 cores of a node would see the same uptime of the server, half of the entries show one time stamp, and the other 28 cores show the other one. That demonstrates that the 56 processes have been running independently on 2 nodes.&lt;br /&gt;
&lt;br /&gt;
===Embedded Job Resource Requests===&lt;br /&gt;
&lt;br /&gt;
The job script can be modified to embed the resource requests in form of a series of &#039;&#039;&#039;#PBS&#039;&#039;&#039; statements at the beginning of the script file. This is a very common practice use at many HPC installations and job submission engines. Let&#039;s go back to the previous example where we run the script on two nodes in parallel. That is the &#039;&#039;&#039;&amp;quot;parallel.job&amp;quot;&#039;&#039;&#039; script file again:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
#!/bin/bash&lt;br /&gt;
#&lt;br /&gt;
#PBS -q xeon28&lt;br /&gt;
#PBS -l walltime=1:0:0&lt;br /&gt;
#PBS -l select=2:ncpus=28:mpiprocs=28:mem=60G&lt;br /&gt;
#&lt;br /&gt;
# the following ensures that you will change into the directory where you are&lt;br /&gt;
# submitting the job from.&lt;br /&gt;
cd $PBS_O_WORKDIR&lt;br /&gt;
#&lt;br /&gt;
# to execute a simple command on all of the cores of all of the nodes allocated to the job,&lt;br /&gt;
# we need to make one of the MPI versions available. Let&#039;s use one of the most up-to-date&lt;br /&gt;
# MPI library available on the cluster&lt;br /&gt;
module load intel/2024.2.0/mpi/2021.13&lt;br /&gt;
#&lt;br /&gt;
# now we are apply a few settings that ensure that the MPI library will use the highest-performing&lt;br /&gt;
# Infiniband Interconnect, as well as a few options to tell MPI how to interface nodes with&lt;br /&gt;
# each other and which specific Infiniband adapter to use. This is complex and requires in-depth&lt;br /&gt;
# knowledge of the QLogic Infiniband adapters we are using. It is unlikely that you will ever have to&lt;br /&gt;
# deal with these options, because the &amp;quot;module load&amp;quot; command for the engineering applications we provide&lt;br /&gt;
# on ARROW will handle all those details transparently without the user needing to understand the details.&lt;br /&gt;
export I_MPI_HYDRA_BOOTSTRAP=ssh&lt;br /&gt;
export MPI_DEVICE=rdma:ofa-v2-ib0&lt;br /&gt;
export UCX_NET_DEVICES=qib0:1&lt;br /&gt;
#&lt;br /&gt;
# it doesn&#039;t make much sense, but in this example we are executing the OS command &amp;quot;uptime&amp;quot; on all cores&lt;br /&gt;
# of the nodes allocated to this job. The output from each core is written to the file info.log. We&lt;br /&gt;
# will find 56 lines of output in the file info.log, each created by the corresponding core executing&lt;br /&gt;
# the uptime command.&lt;br /&gt;
mpirun uptime &amp;gt; info.log&lt;br /&gt;
#&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
If the resource requests are embedded within the file, they don&#039;t have to be specified on the command line any longer (the command line overrides the embedded specifications though). This may be convenient, because all the user has to do for job submission is the following:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
qsub parallel.job&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
Here is an example with more resource specifications and job settings that affect the behavior of the job:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
#!/bin/bash&lt;br /&gt;
#&lt;br /&gt;
#PBS -q xeon28&lt;br /&gt;
#PBS -l walltime=1:0:0&lt;br /&gt;
#PBS -l select=2:ncpus=28:mpiprocs=28:mem=60G&lt;br /&gt;
#PBS -A Account&lt;br /&gt;
#PBS -j oe&lt;br /&gt;
#PBS -N JobName&lt;br /&gt;
#PBS -e log.error&lt;br /&gt;
#PBS -o log.output&lt;br /&gt;
#PBS -m bae&lt;br /&gt;
#&lt;br /&gt;
...&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
I leave this to you as an exercise to figure out what the various options mean and how to specify them. There are many more, all documented in the PBS PRO manual (see above). Most of them are not terribly relevant and can be omitted.&lt;br /&gt;
&lt;br /&gt;
===Interactive Jobs===&lt;br /&gt;
&lt;br /&gt;
On ARROW, we don&#039;t restrict queues to be used only in batch mode. While &#039;&#039;&#039;batch mode&#039;&#039;&#039; is efficient for lining up a lot of work to be executed one after the other, ARROW has been designed to &#039;&#039;&#039;allow efficient model and software development in interactive mode&#039;&#039;&#039;. We have always ensured to have more computers than minimally needed to make it possible to dedicate resources to developers as needed, even if that means wasted CPU cycles. At times, we may ask you to limit the number of interactive jobs so that a large batch workload can be processed efficiently. This happens from time to time, and we have our users coordinate this with each other.&lt;br /&gt;
&lt;br /&gt;
Let&#039;s assume that you are developing an MPI application, or you are working on a complex &#039;&#039;&#039;OpenFOAM&#039;&#039;&#039; model that requires to start parallel processes over and over again just to find a bug and then fix it quickly. To do that, you can &#039;&#039;&#039;request an interactive job&#039;&#039;&#039; by adding the &#039;&#039;&#039;&amp;quot;-I&amp;quot;&#039;&#039;&#039; option to the job submission command (this is an uppercase I). Let&#039;s go to the parallel multi-node example from above:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
qsub -I -q xeon28 -l walltime=1:0:0 -l select=2:ncpus=28:mpiprocs=28:mem=60G&lt;br /&gt;
      ^    ^                  ^               ^       ^           ^      ^ ^&lt;br /&gt;
      |    |                  |               |       |           |      | + --- don&#039;t forget to specify gigabytes&lt;br /&gt;
      |    |                  |               |       |           |      + ----- the amount of memory to request per node&lt;br /&gt;
      |    |                  |               |       |           + ------------ the number of MPI tasks per nodes&lt;br /&gt;
      |    |                  |               |       + ------------------------ the number of cores per node&lt;br /&gt;
      |    |                  |               + -------------------------------- the number of nodes to select in the queue&lt;br /&gt;
      |    |                   + ----------------------------------------------- the requested time, in this case 1h&lt;br /&gt;
      |    + ------------------------------------------------------------------- the queue to be used for the job&lt;br /&gt;
      + ------------------------------------------------------------------------ request an interactive job &amp;lt;&amp;lt;===&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
When running interactive jobs with the &#039;&#039;&#039;&amp;quot;-I&amp;quot;&#039;&#039;&#039; parameter, we don&#039;t specify av job script at the end of the submission command. The interactive job will instead start (once the nodes are available) in interactive mode, meaning that the terminal session changes over from being a series of commands executed on the login server to being a series of commands being executed on the first node of the group of nodes that are allocated to the job. At this point, you can change to the desired working directory, but what you do with the allocated resources is entirely up to you. You can load modules, including MPI libraries, and then issue the commands for your application interactively and see how they execute. If you start an &#039;&#039;&#039;&amp;quot;mpirun&amp;quot;&#039;&#039;&#039;, the cores on your allocated secondary node will work as expected. There is no difference to batch mode, other than you having the ability to execute lines of commands at will.&lt;br /&gt;
&lt;br /&gt;
===Interactive Jobs with X-Windows GUI Applications===&lt;br /&gt;
&lt;br /&gt;
Interactive use can go further than that. With some of our software applications, like &#039;&#039;&#039;StarCCM+&#039;&#039;&#039;, you can run an &#039;&#039;&#039;interactive GUI application&#039;&#039;&#039; where you control the computational work from within the applications&#039; GUI. Within the GUI, you can control execution of the numerical solver that runs on as many cores as you requested, while being able to reconfigure the case through the GUI as well. Furthermore, you can visualize developing results on the fly by creating complex plots and visualizations.&lt;br /&gt;
&lt;br /&gt;
All that is need is an option &#039;&#039;&#039;&amp;quot;-X&amp;quot;&#039;&#039;&#039; being used as part of the job submission, like this:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
qsub -X -I -q xeon28 -l walltime=1:0:0 -l select=2:ncpus=28:mpiprocs=28:mem=60G&lt;br /&gt;
      ^  ^    ^                  ^               ^       ^           ^      ^ ^&lt;br /&gt;
      |  |    |                  |               |       |           |      | + --- don&#039;t forget to specify gigabytes&lt;br /&gt;
      |  |    |                  |               |       |           |      + ----- the amount of memory to request per node&lt;br /&gt;
      |  |    |                  |               |       |           + ------------ the number of MPI tasks per nodes&lt;br /&gt;
      |  |    |                  |               |       + ------------------------ the number of cores per node&lt;br /&gt;
      |  |    |                  |               + -------------------------------- the number of nodes to select in the queue&lt;br /&gt;
      |  |    |                   + ----------------------------------------------- the requested time, in this case 1h&lt;br /&gt;
      |  |    + ------------------------------------------------------------------- the queue to be used for the job&lt;br /&gt;
      |  + ------------------------------------------------------------------------ request an interactive job&lt;br /&gt;
      + --------------------------------------------------------------------------- request GUI capabilities &amp;lt;&amp;lt;===&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
===Running Multiple Jobs on Single Nodes===&lt;br /&gt;
&lt;br /&gt;
A feature that is new on ARROW is the ability to run multiple jobs on a single node. Let&#039;s assume that you are performing a sensitivity analysis on an existing model, and the model is simple enough to return results within a reasonable time on just a few cores of a higher end machine (maybe you are running SMP versions of &#039;&#039;&#039;LS-Dyna&#039;&#039;&#039;). Our high end machines have 64 cores, so lets assume you have an &#039;&#039;&#039;LS-Dyna&#039;&#039;&#039; model that runs well on 8 cores and doesn&#039;t use a whole lot of memory. In this case, you can submit individual jobs that request simply 8 cores and a fraction of the available memory available on the node, and all jobs execute independently from each other. Each job is fit into a slot where available. It is not very different from using whole nodes for everything. The important consideration is that each job is cleanly constrained into it&#039;s allotted resources using the &#039;&#039;&#039;CGROUPS&#039;&#039;&#039; functionality of modern operating systems. Because an abusive user cannot use more cores or more memory than allocated to his job, other users can safely run smaller jobs on the same node.&lt;br /&gt;
&lt;br /&gt;
Lets assume that we have a number of smaller jobs that we want to run on a single node in the &#039;&#039;&#039;&amp;quot;xeon28&amp;quot;&#039;&#039;&#039; queue. Each job would be submitted by using reduced resources that allow for sharing but that  guarantee that the jobs will be run successfully. In this case, you can &#039;&#039;&#039;submit many jobs&#039;&#039;&#039; in the following manner (with a job script for the small jobs, each of which can &#039;&#039;&#039;request varying resources&#039;&#039;&#039; if needed - some may want to run on 5 cores, others on 3):&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
#!/bin/bash&lt;br /&gt;
#&lt;br /&gt;
# the following ensures that you will change into the directory where you are&lt;br /&gt;
# submitting the job from.&lt;br /&gt;
cd $PBS_O_WORKDIR&lt;br /&gt;
#&lt;br /&gt;
# now we sleep for 300 seconds and waste time. This is a placeholder for your application,&lt;br /&gt;
# which would be doing useful work for you.&lt;br /&gt;
sleep 300&lt;br /&gt;
#&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
Now we submit a variety of these jobs (11 total in this example) to the &#039;&#039;&#039;&amp;quot;xeon28&amp;quot;&#039;&#039;&#039; queue for execution (note that the first few jobs request different amounts of memory and core counts):&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
qsub -q xeon28 -l walltime=1:0:0 -l select=1:ncpus=12:mpiprocs=12:mem=5G small.job &lt;br /&gt;
qsub -q xeon28 -l walltime=1:0:0 -l select=1:ncpus=10:mpiprocs=10:mem=7G small.job &lt;br /&gt;
qsub -q xeon28 -l walltime=1:0:0 -l select=1:ncpus=8:mpiprocs=8:mem=9G small.job &lt;br /&gt;
qsub -q xeon28 -l walltime=1:0:0 -l select=1:ncpus=16:mpiprocs=16:mem=20G small.job &lt;br /&gt;
qsub -q xeon28 -l walltime=1:0:0 -l select=1:ncpus=2:mpiprocs=2:mem=2G small.job &lt;br /&gt;
qsub -q xeon28 -l walltime=1:0:0 -l select=1:ncpus=2:mpiprocs=2:mem=2G small.job &lt;br /&gt;
qsub -q xeon28 -l walltime=1:0:0 -l select=1:ncpus=2:mpiprocs=2:mem=2G small.job &lt;br /&gt;
qsub -q xeon28 -l walltime=1:0:0 -l select=1:ncpus=2:mpiprocs=2:mem=2G small.job &lt;br /&gt;
qsub -q xeon28 -l walltime=1:0:0 -l select=1:ncpus=2:mpiprocs=2:mem=2G small.job &lt;br /&gt;
qsub -q xeon28 -l walltime=1:0:0 -l select=1:ncpus=2:mpiprocs=2:mem=2G small.job &lt;br /&gt;
qsub -q xeon28 -l walltime=1:0:0 -l select=1:ncpus=2:mpiprocs=2:mem=2G small.job &lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
They are now running in the order of submission, allocated on as few nodes in the &amp;quot;xeon28&amp;quot; queue as necessary. Only 2 nodes are being loaded quite heavily, and 4 more cores are in use on a third node.&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
Nov 20 23:34 ley@login3:myjobfolder$ qstat -an1&lt;br /&gt;
&lt;br /&gt;
pbs: &lt;br /&gt;
                                                            Req&#039;d  Req&#039;d   Elap&lt;br /&gt;
Job ID          Username Queue    Jobname    SessID NDS TSK Memory Time  S Time&lt;br /&gt;
--------------- -------- -------- ---------- ------ --- --- ------ ----- - -----&lt;br /&gt;
&lt;br /&gt;
3082.pbs        ley      xeon28   small.job  813221   1  12    5gb 01:00 R 00:01 p001/0*12&lt;br /&gt;
3083.pbs        ley      xeon28   small.job  813288   1  10    7gb 01:00 R 00:01 p001/1*10&lt;br /&gt;
3084.pbs        ley      xeon28   small.job  671792   1   8    9gb 01:00 R 00:01 p002/0*8&lt;br /&gt;
3085.pbs        ley      xeon28   small.job  671845   1  16   20gb 01:00 R 00:01 p002/1*16&lt;br /&gt;
3086.pbs        ley      xeon28   small.job  813361   1   2    2gb 01:00 R 00:00 p001/2*2&lt;br /&gt;
3087.pbs        ley      xeon28   small.job  813413   1   2    2gb 01:00 R 00:00 p001/3*2&lt;br /&gt;
3088.pbs        ley      xeon28   small.job  813464   1   2    2gb 01:00 R 00:00 p001/4*2&lt;br /&gt;
3089.pbs        ley      xeon28   small.job  671912   1   2    2gb 01:00 R 00:00 p002/2*2&lt;br /&gt;
3090.pbs        ley      xeon28   small.job  671969   1   2    2gb 01:00 R 00:00 p002/3*2&lt;br /&gt;
3091.pbs        ley      xeon28   small.job  632092   1   2    2gb 01:00 R 00:00 p003/0*2&lt;br /&gt;
3092.pbs        ley      xeon28   small.job  632100   1   2    2gb 01:00 R 00:00 p003/1*2&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
This is a particularly effective strategy to run concurrently many cases that don&#039;t scale well beyond a few cores. When running them on fewer cores but many of them at the same time, the overall processing rate will be much higher than executing them the traditional way.&lt;br /&gt;
&lt;br /&gt;
===Running Jobs using GPUs===&lt;br /&gt;
&lt;br /&gt;
The principle of running multiple jobs on a single node becomes particularly important when using servers equipped with &#039;&#039;&#039;GPUs&#039;&#039;&#039; for &#039;&#039;&#039;ML/AI&#039;&#039;&#039; applications. The cluster doesn&#039;t have a whole lot of &#039;&#039;&#039;GPUs&#039;&#039;&#039; at this point. We have three machines with three &#039;&#039;&#039;A4000&#039;&#039;&#039; GOUs, a &#039;&#039;&#039;total of 9 A4000 GPUs&#039;&#039;&#039;. Then we have a much more powerful single machine with our &#039;&#039;&#039;four A6000 GPUs&#039;&#039;&#039;.&lt;br /&gt;
&lt;br /&gt;
Using multiple GPUs in a single application is still something where the software has to be designed with hardware in mind. GPUs have several methods of communicating with each other, e.g. very fast &#039;&#039;&#039;NVLINK&#039;&#039;&#039; between pairs of GPUs, GPUs being directly connected to one of the two CPUs in the system and thus being able to communicate faster, and GPUs that have to jump between processors when communicating, and then the whole issue of having to go possibly through PCIe bridges.&lt;br /&gt;
&lt;br /&gt;
On our system, we are providing the ability to &#039;&#039;&#039;work mostly with individual GPUs&#039;&#039;&#039;. Users can also reserve entire nodes and develop or run applications that are adapted to that hardware, including several GPUs installed on that node. One thing we do not provide is the ability of GPU to GPU communication between nodes. Thus, a job cannot request more than one GPU node at a time.&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
qsub -q a4000 -I -l walltime=1:0:0 -l select=1:ncpus=5:mem=150G:ngpus=1&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
With these specifications, three single GPU jobs can run on a single server. Each job sees only one of the reserved GPU.&lt;br /&gt;
&lt;br /&gt;
To run a massive GPU job on 64 cores with 4 &#039;&#039;&#039;A6000 GPUs&#039;&#039;&#039;, submit the job like this:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
qsub -q a6000 -I -l walltime=1:0:0 -l select=1:ncpus=64:mem=725G:ngpus=4&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
===Manual pages for qsub===&lt;br /&gt;
&lt;br /&gt;
To learn more about the &amp;quot;qsub&amp;quot; command, you can use the command &amp;quot;man qsub&amp;quot;, which will print a lot of detailed information about the capabilities of this command.&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
$ man qsub&lt;br /&gt;
&lt;br /&gt;
qsub(1B)                                               PBS Professional                                              qsub(1B)&lt;br /&gt;
&lt;br /&gt;
NAME&lt;br /&gt;
       qsub - submit a job to PBS&lt;br /&gt;
&lt;br /&gt;
SYNOPSIS&lt;br /&gt;
       qsub [-a &amp;lt;date and time&amp;gt;] [-A &amp;lt;account string&amp;gt;] [-c &amp;lt;checkpoint spec&amp;gt;]&lt;br /&gt;
            [-C &amp;lt;directive prefix&amp;gt;] [-e &amp;lt;path&amp;gt;] [-f] [-h]&lt;br /&gt;
            [-I [-G [-- &amp;lt;GUI application/script&amp;gt;]] | [-X]] [-j &amp;lt;join&amp;gt;]&lt;br /&gt;
            [-J &amp;lt;range&amp;gt; [%&amp;lt;max subjobs]] [-k &amp;lt;discard&amp;gt;] [-l &amp;lt;resource list&amp;gt;]&lt;br /&gt;
            [-m &amp;lt;mail events&amp;gt;] [-M &amp;lt;user list&amp;gt;] [-N &amp;lt;name&amp;gt;] [-o &amp;lt;path&amp;gt;]&lt;br /&gt;
            [-p &amp;lt;priority&amp;gt;] [-P &amp;lt;project&amp;gt;] [-q &amp;lt;destination&amp;gt;] [-r &amp;lt;y|n&amp;gt;]&lt;br /&gt;
            [-R &amp;lt;remove options&amp;gt;] [-S &amp;lt;path list&amp;gt;] [-u &amp;lt;user list&amp;gt;]&lt;br /&gt;
            [-v &amp;lt;variable list&amp;gt;] [-V] [-W &amp;lt;additional attributes&amp;gt;] [-z]&lt;br /&gt;
            [- | &amp;lt;script&amp;gt; | -- &amp;lt;executable&amp;gt; [&amp;lt;arguments to executable&amp;gt;]]&lt;br /&gt;
       qsub --version&lt;br /&gt;
&lt;br /&gt;
DESCRIPTION&lt;br /&gt;
       You use the qsub command to submit a batch job to PBS.  Submitting a PBS job specifies a task, requests resources, and&lt;br /&gt;
       sets job attributes.&lt;br /&gt;
... &amp;lt;many more pages&amp;gt;&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
==LS-Dyna on the ARROW Cluster==&lt;br /&gt;
&lt;br /&gt;
===Currently Available LS-Dyna Versions===&lt;br /&gt;
&lt;br /&gt;
The following is a list of &#039;&#039;&#039;LS-Dyna versions&#039;&#039;&#039; available on &#039;&#039;&#039;ARROW&#039;&#039;&#039; after the latest reconfiguration of the system. As per LSTC/ANSYS, &#039;&#039;&#039;versions before 14.0.0 are not necessarily fully supported any longer&#039;&#039;&#039; because they are supposedly not compatible with modern operating systems and cannot be made to work reliably. We have tested the listed older versions of LS-Dyna and they have passed basic tests. They may not behave exactly as they did on the old CentOS 7 operating system, and time will show whether they can still be used or whether you will need to update your models and use a fully supported version.&lt;br /&gt;
&lt;br /&gt;
All versions are loaded using the &#039;&#039;&#039;&amp;quot;module load&amp;quot;&#039;&#039;&#039; command. Versions can be listed with the &#039;&#039;&#039;&amp;quot;module avail ls-dyna&amp;quot;&#039;&#039;&#039; command. To load one of the modules, use the following syntax:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
module load ls-dyna/14.2.0/mpi-d8-ifort190-avx512&lt;br /&gt;
            ^       ^      ^   ^  ^        ^&lt;br /&gt;
            |       |      |   |  |        + --- specify the extended instruction set needed for execution&lt;br /&gt;
            |       |      |   |  + ------------ load the version of the compiler that was used to create this &lt;br /&gt;
            |       |      |   + --------------- load the version that supports double precision variables&lt;br /&gt;
            |       |      + ------------------- load the MPP (MPI) version of LS-Dyna&lt;br /&gt;
            |       + -------------------------- load specifically version 14.2.0&lt;br /&gt;
            + ---------------------------------- load a version of LS-Dyna&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
The version string is composed of multiple elements to indicate variants in compilers and compiler options. Use the following guideline to choose an appropriate version to load:&lt;br /&gt;
&lt;br /&gt;
* &#039;&#039;&#039;&amp;quot;1&amp;quot;&#039;&#039;&#039; or &#039;&#039;&#039;&amp;quot;mpi&amp;quot;&#039;&#039;&#039; indicates whether this is a single node version of LS-Dyna (&#039;&#039;&#039;SMP&#039;&#039;&#039;) or whether this is a multi-node MPI version (&#039;&#039;&#039;MPP&#039;&#039;&#039;). All MPI versions use the &#039;&#039;&#039;IntelMPI 2022&#039;&#039;&#039; libraries which have been tested thoroughly on ARROW. MPI versions will use the Infiniband Network of ARROW for high-speed and low-latency inter-process communication using RDMA (remote direct memory access).&lt;br /&gt;
* All LS-Dyna versions are available in either &#039;&#039;&#039;floating point&#039;&#039;&#039; or &#039;&#039;&#039;double precision variants&#039;&#039;&#039;. Floating point variants use 4 bytes to represent a value, and double precision variants use 8 bytes. There are pros and cons for choosing one over the other variant. With regards to computational efficiency, both perform nearly the same because all machines are equipped with 64-bit CPUs.&lt;br /&gt;
** &#039;&#039;&#039;&amp;quot;f4&amp;quot;&#039;&#039;&#039; floating point versions&lt;br /&gt;
*** Pros: These require significantly less memory to run. Results occupy less disk space, and can be transferred significantly faster into and out of ARROW.&lt;br /&gt;
*** Cons: The numerical resolution is limited to 7 significant digits, which is often undesirable when dealing with mathematical operations on small and large numbers at the same time.&lt;br /&gt;
** &#039;&#039;&#039;&amp;quot;r8&amp;quot;&#039;&#039;&#039; double precision versions&lt;br /&gt;
*** Pros: The numerical resolution is about twice the number of significant digits compare to &amp;quot;f4&amp;quot;, which helps when when dealing with mathematical operations on small and large numbers at the same time.&lt;br /&gt;
*** Cons: These require more memory to run. Results occupy more disk space, and it takes longer to transfer data into and out of ARROW.&lt;br /&gt;
* There are two more identifiers to choose from when it comes to the variants of the executables: the specific compiler used to create the executable and the specific processor instruction set required for running the executable.&lt;br /&gt;
** For modern versions of LS-Dyna, two compilers have been used by the developers to create LS-Dyna executables: the &#039;&#039;&#039;Intel Fortran Compiler&#039;&#039;&#039; and the &#039;&#039;&#039;AOCC (AMD Optimizing C/C++ and Fortran) compiler&#039;&#039;&#039;. Both variants of the software are supported on ARROW. This gives users the opportunity to choose an alternate variant of the same LS-Dyna version when running into bugs or crashes.&lt;br /&gt;
** The variants based on the various instruction set extensions (&#039;&#039;&#039;SSE2&#039;&#039;&#039;, &#039;&#039;&#039;AVX2&#039;&#039;&#039;, &#039;&#039;&#039;AVX512&#039;&#039;&#039;, and so on) gives users even more options when choosing an alternate LS-Dyna variant of the same version when running into bugs or crashes. These instruction sets are mostly related to performance gains on specific processors. We have not performed thorough performance tests and cannot recommend specific versions right now.&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
$ module avail ls-dyna&lt;br /&gt;
--------------------------------------------- /shared/apps/modulefiles ---------------------------------------------&lt;br /&gt;
ls-dyna/09.3.1/1-d8-ifort131           ls-dyna/12.2.1/mpi-f4-ifort160-sse2    ls-dyna/14.2.0/mpi-f4-ifort190-avx512  &lt;br /&gt;
ls-dyna/09.3.1/1-f4-ifort131           ls-dyna/12.2.2/1-d8-ifort160-sse2      ls-dyna/14.2.0/mpi-f4-ifort190-sse2    &lt;br /&gt;
ls-dyna/09.3.1/mpi-d8-ifort131-avx2    ls-dyna/12.2.2/1-f4-ifort160-sse2      ls-dyna/15.0.2/1-d8-ifort190-sse2      &lt;br /&gt;
ls-dyna/09.3.1/mpi-d8-ifort131-avx512  ls-dyna/12.2.2/mpi-d8-aocc400-avx2     ls-dyna/15.0.2/1-f4-ifort190-sse2      &lt;br /&gt;
ls-dyna/09.3.1/mpi-f4-ifort131-avx2    ls-dyna/12.2.2/mpi-d8-ifort160-avx2    ls-dyna/15.0.2/mpi-d8-aocc400-avx2     &lt;br /&gt;
ls-dyna/09.3.1/mpi-f4-ifort131-avx512  ls-dyna/12.2.2/mpi-d8-ifort160-sse2    ls-dyna/15.0.2/mpi-d8-ifort190-avx2    &lt;br /&gt;
ls-dyna/10.2.0/1-d8-ifort160           ls-dyna/12.2.2/mpi-f4-aocc400-avx2     ls-dyna/15.0.2/mpi-d8-ifort190-avx512  &lt;br /&gt;
ls-dyna/10.2.0/1-f4-ifort160           ls-dyna/12.2.2/mpi-f4-ifort160-avx2    ls-dyna/15.0.2/mpi-d8-ifort190-sse2    &lt;br /&gt;
ls-dyna/11.0.0/1-d8-ifort160           ls-dyna/12.2.2/mpi-f4-ifort160-sse2    ls-dyna/15.0.2/mpi-f4-aocc400-avx2     &lt;br /&gt;
ls-dyna/11.0.0/1-f4-ifort160           ls-dyna/13.0.0/1-d8-ifort190           ls-dyna/15.0.2/mpi-f4-ifort190-avx2    &lt;br /&gt;
ls-dyna/11.1.0/1-d8-ifort160-sse2      ls-dyna/13.0.0/1-f4-ifort190           ls-dyna/15.0.2/mpi-f4-ifort190-avx512  &lt;br /&gt;
ls-dyna/11.1.0/1-f4-ifort160-sse2      ls-dyna/13.0.0/mpi-d8-ifort190-avx2    ls-dyna/15.0.2/mpi-f4-ifort190-sse2    &lt;br /&gt;
ls-dyna/11.2.0/1-d8-ifort160           ls-dyna/13.0.0/mpi-d8-ifort190-sse2    ls-dyna/16.0.0/1-d8-aocc420-avx2       &lt;br /&gt;
ls-dyna/11.2.0/1-f4-ifort160           ls-dyna/13.0.0/mpi-f4-ifort190-avx2    ls-dyna/16.0.0/1-d8-aocc420-avx512     &lt;br /&gt;
ls-dyna/11.2.0/mpi-f4-ifort160-avx2    ls-dyna/13.0.0/mpi-f4-ifort190-sse2    ls-dyna/16.0.0/1-d8-ifort190-sse2      &lt;br /&gt;
ls-dyna/11.2.0/mpi-f4-ifort160-sse2    ls-dyna/13.1.0/mpi-d8-aocc310-avx2     ls-dyna/16.0.0/1-f4-aocc420-avx2       &lt;br /&gt;
ls-dyna/11.2.1/1-d8-ifort160           ls-dyna/13.1.0/mpi-d8-ifort190-avx2    ls-dyna/16.0.0/1-f4-aocc420-avx512     &lt;br /&gt;
ls-dyna/11.2.1/1-f4-ifort160           ls-dyna/13.1.0/mpi-d8-ifort190-sse2    ls-dyna/16.0.0/1-f4-ifort190-sse2      &lt;br /&gt;
ls-dyna/11.2.1/mpi-d8-ifort160-avx2    ls-dyna/13.1.0/mpi-f4-aocc310-avx2     ls-dyna/16.0.0/mpi-d8-aocc420-avx2     &lt;br /&gt;
ls-dyna/11.2.1/mpi-d8-ifort160-sse2    ls-dyna/13.1.0/mpi-f4-ifort190-avx2    ls-dyna/16.0.0/mpi-d8-aocc420-avx512   &lt;br /&gt;
ls-dyna/11.2.1/mpi-f4-ifort160-avx2    ls-dyna/13.1.0/mpi-f4-ifort190-sse2    ls-dyna/16.0.0/mpi-d8-ifort190-avx2    &lt;br /&gt;
ls-dyna/11.2.1/mpi-f4-ifort160-sse2    ls-dyna/13.1.1/mpi-d8-ifort190-avx2    ls-dyna/16.0.0/mpi-d8-ifort190-avx512  &lt;br /&gt;
ls-dyna/11.2.2/mpi-d8-ifort160-avx2    ls-dyna/13.1.1/mpi-d8-ifort190-sse2    ls-dyna/16.0.0/mpi-d8-ifort190-sse2    &lt;br /&gt;
ls-dyna/11.2.2/mpi-d8-ifort160-sse2    ls-dyna/13.1.1/mpi-f4-ifort190-avx2    ls-dyna/16.0.0/mpi-f4-aocc420-avx2     &lt;br /&gt;
ls-dyna/11.2.2/mpi-f4-ifort160-avx2    ls-dyna/13.1.1/mpi-f4-ifort190-sse2    ls-dyna/16.0.0/mpi-f4-aocc420-avx512   &lt;br /&gt;
ls-dyna/11.2.2/mpi-f4-ifort160-sse2    ls-dyna/14.0.0/1-d8-ifort190           ls-dyna/16.0.0/mpi-f4-ifort190-avx2    &lt;br /&gt;
ls-dyna/12.1.0/1-d8-ifort160           ls-dyna/14.0.0/1-f4-ifort190           ls-dyna/16.0.0/mpi-f4-ifort190-avx512  &lt;br /&gt;
ls-dyna/12.1.0/1-f4-aocc310            ls-dyna/14.0.0/mpi-d8-aocc310-avx2     ls-dyna/16.0.0/mpi-f4-ifort190-sse2    &lt;br /&gt;
ls-dyna/12.1.0/1-f4-ifort160           ls-dyna/14.0.0/mpi-d8-ifort190-avx2    ls-dyna/16.1.0/mpi-d8-aocc420-avx2     &lt;br /&gt;
ls-dyna/12.1.0/mpi-d8-aocc310-avx2     ls-dyna/14.0.0/mpi-d8-ifort190-sse2    ls-dyna/16.1.0/mpi-d8-aocc420-avx512   &lt;br /&gt;
ls-dyna/12.1.0/mpi-d8-ifort160-avx2    ls-dyna/14.0.0/mpi-f4-ifort190-avx2    ls-dyna/16.1.0/mpi-d8-ifort190-avx2    &lt;br /&gt;
ls-dyna/12.1.0/mpi-d8-ifort160-sse2    ls-dyna/14.0.0/mpi-f4-ifort190-sse2    ls-dyna/16.1.0/mpi-d8-ifort190-avx512  &lt;br /&gt;
ls-dyna/12.1.0/mpi-f4-aocc310-avx2     ls-dyna/14.1.0/1-d8-ifort190-sse2      ls-dyna/16.1.0/mpi-d8-ifort190-sse2    &lt;br /&gt;
ls-dyna/12.1.0/mpi-f4-ifort160-avx2    ls-dyna/14.1.0/1-f4-ifort190-sse2      ls-dyna/16.1.0/mpi-f4-aocc420-avx2     &lt;br /&gt;
ls-dyna/12.1.0/mpi-f4-ifort160-sse2    ls-dyna/14.1.0/mpi-d8-aocc400-avx2     ls-dyna/16.1.0/mpi-f4-aocc420-avx512&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
===Submitting an LS-Dyna Job===&lt;br /&gt;
&lt;br /&gt;
&amp;lt;BLOCKQUOTE&amp;gt;&lt;br /&gt;
[[File:Attention.jpg|25px]] &#039;&#039;&#039;IMPORTANT NOTE:&#039;&#039;&#039; &#039;&#039;The job/queue manager can track the number of LS-Dyna licenses to some degree. If all &#039;&#039;&#039;LS-Dyna users&#039;&#039;&#039; cooperate and use a script like the one shown below when submitting their jobs, the total number of concurrent &#039;&#039;&#039;LS-Dyna licenses&#039;&#039;&#039; will be tracked by the job manager correctly. That means that users can submit any number of LS-Dyna jobs, and jobs will only start when a sufficient number of licenses is available. This is managed by the &#039;&#039;&#039;&amp;quot;dynalic&amp;quot;&#039;&#039;&#039; resource at the end of the select statement. In this example, a 2-node job on 64-core nodes will need a total of &#039;&#039;&#039;&amp;quot;dynalic=128&amp;quot;&#039;&#039;&#039; licenses. This accounting breaks down when users don&#039;t use the &#039;&#039;&#039;&amp;quot;dynalic=XXX&amp;quot;&#039;&#039;&#039; statement, or when they don&#039;t calculate the number of licenses correctly. In that case, LS-Dyna jobs of all users are subject to sudden failure when LS-Dyna licenses run out. Please understand the importance of this specific setting in your job.&#039;&#039;&lt;br /&gt;
&amp;lt;/BLOCKQUOTE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
Furthermore, careful consideration should be given with regards to choice of resources for an &#039;&#039;&#039;LS-Dyna job&#039;&#039;&#039;. With 64 cores available on a single node in the &#039;&#039;&#039;&amp;quot;epyc1&amp;quot;&#039;&#039;&#039; and &#039;&#039;&#039;&amp;quot;epyc2&amp;quot;&#039;&#039;&#039; queues, it may be counterproductive to run a job on two nodes instead of a single node. Users should run their jobs with different numbers of nodes and determine whether performance increases. It may well decrease when running a job on two or more nodes. The outcome of such tests will tell what the best allocation of resources will be.&lt;br /&gt;
&lt;br /&gt;
Most users use a job script like the following. All methods for job submission the the previous chapters apply as well, so there is a lot of flexibility:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
#!/bin/bash&lt;br /&gt;
#&lt;br /&gt;
#PBS -q epyc1&lt;br /&gt;
#PBS -l walltime=12:0:0&lt;br /&gt;
#PBS -l select=2:ncpus=64:mpiprocs=64:mem=225G,dynalic=128&lt;br /&gt;
#PBS -N JobName&lt;br /&gt;
#PBS -e log.error&lt;br /&gt;
#PBS -o log.output&lt;br /&gt;
#&lt;br /&gt;
cd $PBS_O_WORKDIR&lt;br /&gt;
#&lt;br /&gt;
module load ls-dyna/12.2.1/mpi-f4-ifort160-avx2&lt;br /&gt;
module load dynamore/current&lt;br /&gt;
#&lt;br /&gt;
mpirun ls-dyna i=main.k memory1=300m memory2=100m &amp;gt; dyna.log&lt;br /&gt;
#&lt;br /&gt;
# when using the Dynamore tools, you can start something like this at the end&lt;br /&gt;
DM.plotcprs.lnx -merge &amp;gt;&amp;gt; dyna.log&lt;br /&gt;
#&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
===LSTC Tools: LS-OPT and LS-PREPOST===&lt;br /&gt;
&lt;br /&gt;
For the new Rocky 9 cluster, I have not looked deeply into the ls-opt and ls-prepost packages that were installed. I noticed though that the LSTC server provided access to much newer versions of both software packages. If you would like to learn more or have a specific version in mind, I can easily download and install it for you.&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
$ module avail ls-opt&lt;br /&gt;
----------------------------------------------- /shared/apps/modulefiles ------------------------------------------------&lt;br /&gt;
ls-opt/5.1.1  ls-opt/6.0.0  ls-opt/7.0.0  ls-opt/7.0.2   ls-opt/2022R2  &lt;br /&gt;
ls-opt/5.2.1  ls-opt/6.1.0  ls-opt/7.0.1  ls-opt/2022R1  ls-opt/2023R1  &lt;br /&gt;
&lt;br /&gt;
$ module avail ls-prepost&lt;br /&gt;
----------------------------------------------- /shared/apps/modulefiles ------------------------------------------------&lt;br /&gt;
ls-prepost/4.5.10  ls-prepost/4.8.13  ls-prepost/4.8.30  ls-prepost/4.9.16  ls-prepost/4.10.7 &lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
===Dynamore Software===&lt;br /&gt;
&lt;br /&gt;
The Dynamore tools are available as a module:&lt;br /&gt;
&lt;br /&gt;
 module load dynamore/current&lt;br /&gt;
&lt;br /&gt;
We typically acquire a yearly license for the tools as we purchase licenses for LS-Dyna.&lt;br /&gt;
&lt;br /&gt;
===Vendor License File Installation===&lt;br /&gt;
&lt;br /&gt;
If you would like for us to install a vendor license for LS-Dyna models, please contact us for the required information. We can send you the general LS-Dyna license file to show the host ids for the license server. Using that information, your vendor should be able to create a vendor license file. Please send that file to us per Email or by other means.&lt;br /&gt;
&lt;br /&gt;
==StarCCM+ on the ARROW Cluster==&lt;br /&gt;
&lt;br /&gt;
===Currently Available StarCCM+ Versions===&lt;br /&gt;
&lt;br /&gt;
As of late 2025, we have the following versions of &#039;&#039;&#039;StarCCM+&#039;&#039;&#039; available on the cluster:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
$ module avail starccm&lt;br /&gt;
---------------------------- /shared/apps/modulefiles ----------------------------&lt;br /&gt;
starccm/15.02.007-R8  starccm/16.06.008-R8  starccm/18.06.006-R8  &lt;br /&gt;
starccm/15.02.009-R8  starccm/17.02.007-R8  starccm/19.02.009-R8  &lt;br /&gt;
starccm/15.04.008-R8  starccm/17.02.008-R8  starccm/20.04.007-R8  &lt;br /&gt;
starccm/15.06.008-R8  starccm/17.04.007-R8  starccm/20.06.007-R8  &lt;br /&gt;
starccm/16.02.008-R8  starccm/17.06.007-R8  &lt;br /&gt;
starccm/16.04.007-R8  starccm/18.04.008-R8&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
If using a &#039;&#039;&#039;single node&#039;&#039;&#039; for StarCCM+, job submission (for an interactive job) is simple and will use appropriate default settings:&lt;br /&gt;
&lt;br /&gt;
 qsub -I -X -q epyc1 -l walltime=20:00:00&lt;br /&gt;
&lt;br /&gt;
StarCCM+ can make use of the job scheduler attributes by automatically obtaining the number of cores and other resources from OpenPBS. In this case, the default number of cores and mpi processes for StarCCM+ are both 64 when using the epyc1 queue. So you can start your StarCCM+ run with:&lt;br /&gt;
&lt;br /&gt;
 module load starccm/15.02.007-R8 (or any other version)&lt;br /&gt;
 starccm+ -bs pbs&lt;br /&gt;
&lt;br /&gt;
In this case, there is no need for StarCCM+ to be told to run the case in parallel with the selected number of cores/mpiprocs.&lt;br /&gt;
&lt;br /&gt;
This can get a bit more complex when running on multiple nodes or when requesting high memory nodes. In that case you would use job submission parameters as shown below:&lt;br /&gt;
&lt;br /&gt;
 qsub -I -X -q epyc1 -l walltime=20:00:00,select=2:ncpus=64:mpiprocs=61:mem=500GB&lt;br /&gt;
&lt;br /&gt;
Requesting nodes that can satisfy those resources, two nodes with these attributes must exist. We have multiple nodes with 512GB in the epyc1 queue, meaning that this job will run on two machines that have at least the required amount of memory installed (on each node). The job will be queued until two machines like this will be available. If no machines with these resources exist, the job will stay in the queue forever. Therefore, you have to craft the submission string carefully.&lt;br /&gt;
&lt;br /&gt;
To accommodate high memory jobs, the nodes have been assigned priorities for assignment. Low memory jobs have the highest priority and will be assigned to nodes that can accommodate the request. High memory nodes have the lowest priority, meaning that they are the last ones given out to users. This makes it more likely that a high memory job can be run soon when the cluster is moderately loaded with jobs.&lt;br /&gt;
&lt;br /&gt;
StarCCM+ will always use the Intel MPI fabric. Other MPI versions do not work, even when selected on the command line.&lt;br /&gt;
&lt;br /&gt;
==OpenFOAM on the ARROW Cluster==&lt;br /&gt;
&lt;br /&gt;
===Currently Available OpenFOAM  Versions===&lt;br /&gt;
&lt;br /&gt;
As of late 2025, we have the following versions of OpenFOAM available on the cluster:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
$ module avail openfoam&lt;br /&gt;
------------ /shared/apps/modulefiles ------------&lt;br /&gt;
openfoam/9   openfoam/13      openfoam/v2312  &lt;br /&gt;
openfoam/10  openfoam/13-amd  openfoam/v2406  &lt;br /&gt;
openfoam/11  openfoam/v2212   &lt;br /&gt;
openfoam/12  openfoam/v2306&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
Contact us if you encounter problems; there can be various reasons why OpenFOAM may have trouble on certain hardware or when compiling dynamic code. When loading OpenFOAM modules, a number of dependencies will be automatically loaded for you, and you don&#039;t have to load those yourself. For example:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
$ module load openfoam/13&lt;br /&gt;
Loading openfoam/13&lt;br /&gt;
  Loading requirement: intel/2024.2.0/mpi/2021.13 gcc/gcc-12.1.0&lt;br /&gt;
&lt;br /&gt;
$ module list&lt;br /&gt;
Currently Loaded Modulefiles:&lt;br /&gt;
 1) intel/2024.2.0/mpi/2021.13   2) gcc/gcc-12.1.0   3) openfoam/13&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
In this case, OpenFOAM 13 loads the Intel 2024 MPI module, and loads the GCC compiler 12.1. OpenFOAM was compiled from source, and has been compiled specifically with that compiler and MPI version, so it make little sense to use other compilers or MPI libraries.&lt;br /&gt;
&lt;br /&gt;
Note: We have found a problem with running the Intel 2024 MPI library in the amd64 queue. Therefore, we have a modified module that uses the Intel 2022 library (I know -- Intel 2022 gives you the 2021 MPI libraries, but that is the way Intel distributes this software):&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
$ module load openfoam/13-amd &lt;br /&gt;
Loading mpi version 2021.7.0&lt;br /&gt;
Loading openfoam/13-amd&lt;br /&gt;
  Loading requirement: intel/2022.2.0/mpi/2021.7.0 gcc/gcc-12.1.0&lt;br /&gt;
&lt;br /&gt;
$ module list&lt;br /&gt;
Currently Loaded Modulefiles:&lt;br /&gt;
 1) intel/2022.2.0/mpi/2021.7.0   2) gcc/gcc-12.1.0   3) openfoam/13-amd&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
If you are compiling OpenFOAM yourself, the modules are of little help. You would need to select the appropriate MPI version and compiler before doing so, and then consistently load them before running your OpenFOAM executables. Within the &amp;quot;etc/bashrc&amp;quot; file in the source code tree, you want to set the MPI library to INTELMPI. As usual with self-compiled versions of OpenFOAM, you would &amp;quot;source etc/bashrc&amp;quot; to set up your personal environment to run your home-brew version of OpenFOAM. Contact us if you need to learn more about compiling OpenFOAM on the system.&lt;br /&gt;
&lt;br /&gt;
==Additional Software Applications and Libraries==&lt;br /&gt;
&lt;br /&gt;
===Loadable GCC Compiler Versions===&lt;br /&gt;
&lt;br /&gt;
The Rocky 9.6 operating system uses the GCC 11.5 compiler. That should be sufficient for most users when compiling your own applications. In case you need to use either a more up-to-date compiler, or if you need an older compiler for compatibility, we make the following versions available as loadable modules.&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
$ module avail gcc&lt;br /&gt;
------------ /shared/apps/modulefiles ------------&lt;br /&gt;
gcc/gcc-4.9.4  gcc/gcc-7.5.0  gcc/gcc-10.3.0  &lt;br /&gt;
gcc/gcc-5.5.0  gcc/gcc-8.5.0  gcc/gcc-11.3.0  &lt;br /&gt;
gcc/gcc-6.5.0  gcc/gcc-9.5.0  gcc/gcc-12.1.0&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
Additional versions can be created and made available as modules as well. If you need a specific version that is not currently available, please ask us to compiler and install it. If necessary, we may be able to provide access to other compilers, for example LLVM. We do not provide access to proprietary compilers at this time.&lt;br /&gt;
&lt;br /&gt;
===MPI Libraries and Runtimes===&lt;br /&gt;
&lt;br /&gt;
While we seem to have a variety of MPI versions and flavors available to users, the only MPI versions that allow us to run software over Infiniband are the Intel MPI libraries. Some of the installed alternatives are likely to fail, or will have a set of environment variables that have to be set. All major engineering software packages that we offer are pre-configured with specific MPI versions and settings that have been tested and/or provided by the vendors.&lt;br /&gt;
&lt;br /&gt;
Note: Some MPI libraries may seem to work. They may allow your MPI application to run. But inter-process network communication may travel through the rather slow and high-latency Ethernet fabric, making MPI applications very ineffective and are probably not worth while.&lt;br /&gt;
&lt;br /&gt;
===MatLab Runtimes===&lt;br /&gt;
&lt;br /&gt;
We can install MatLAB run time libraries as needed and have them available as loadable modules. Recently, we had a problem with MatLAB run time libraries being identified as security vulnerabilities. Contact us if you need them installed for one of your projects.&lt;br /&gt;
&lt;br /&gt;
===Anaconda and variants (miniconda etc)===&lt;br /&gt;
&lt;br /&gt;
Our current practice is to have users download and install their own versions of Anaconda and its variants in their own home directories. This allows for maximum flexibility when it comes to installable software modules, and users can maintain the installation, upgrades, and maintenance themselves. If you encounter issues, please contact us. One known side effect of Anaconda installations is a performance hit when starting your software, e.g. python scripts may take 30 seconds or more to execute. This is an artefact caused by the Lustre file system, which has been designed for large files accessible from many machines simultaneously. Performance on reading many small files has not been considered and is fairly poor. Again, contact us and we will design a solution for you as needed.&lt;/div&gt;</summary>
		<author><name>Ley</name></author>
	</entry>
	<entry>
		<id>https://wiki.anl.gov/wiki_tracc/index.php?title=Job_Submission_and_Monitoring&amp;diff=2485</id>
		<title>Job Submission and Monitoring</title>
		<link rel="alternate" type="text/html" href="https://wiki.anl.gov/wiki_tracc/index.php?title=Job_Submission_and_Monitoring&amp;diff=2485"/>
		<updated>2026-02-18T21:13:48Z</updated>

		<summary type="html">&lt;p&gt;Ley: /* Submitting an LS-Dyna Job */&lt;/p&gt;
&lt;hr /&gt;
&lt;div&gt;{| align=&amp;quot;right&amp;quot;&lt;br /&gt;
| __TOC__&lt;br /&gt;
|}&lt;br /&gt;
==Resource Summary View==&lt;br /&gt;
&lt;br /&gt;
To get started, users can query the overall status of resources on the cluster. The &#039;&#039;&#039;&amp;quot;qsum&amp;quot;&#039;&#039;&#039; script will list all queues and nodes, as well as how many are offline, down, free, or assigned to users. This is a script developed by our team, and may need to be updated if something goes wrong. Please contact us if you experience any problems.&lt;br /&gt;
&lt;br /&gt;
Each queue groups a number of nodes together based on their hardware and software configurations. Nodes can be part of more than one queue, and there are other complex details that we are ignoring here for the purpose of keeping it simple.&lt;br /&gt;
&lt;br /&gt;
===Queues===&lt;br /&gt;
&lt;br /&gt;
Here is a very brief summary of what each of the queues is, and how to use them efficiently:&lt;br /&gt;
&lt;br /&gt;
; a4000: This is a queue that has three 16-core CPU machines, each of which is furthermore equipped with three A4000 GPUs. That makes a total of 9 A4000 GPUs available to users. Neither the GPUs nor the processors are particularly powerful these days. The machines have 500GB of memory though, which makes for a good platform for experimenting with GPU capabilities.&lt;br /&gt;
; a6000: This is a queue that has only one 64-core CPU machines, and is equipped with four A6000 GPUs. The system can be upgraded to 8 A6000 GPUs if needed. This is a decent GPU machine that can take a solid workload these days. The machine has 750GB of memory, which makes for a good production platform.&lt;br /&gt;
; amd16: This is a queue with many of our older AMD-based 16-core machines, each of which has 30GB of memory. While individual machines are a bit outdated, they are all interconnected with Infiniband and can provide a solid production workload in multi-nodes jobs over MPI without blocking the more current (and thus expensive) systems.&lt;br /&gt;
; epyc1/epyc2: These are 2 separate queues with slightly different performance characteristics. Each of the groups is interconnected with Infiniband to provide a platform for large and demanding software packages, such as LS-Dyna and StarCCM+. They have between 250GB and 500GB of memory. Because licenses for these software packages are very expensive, they should use these two queues for making optimum use of limited core licenses available to each package.&lt;br /&gt;
; xeon28: This is a set of intermediate machines with 28 cores and 64GB of memory. They can be used for a variety of purposes, including MPI jobs and single node application software.&lt;br /&gt;
; virtual: This is a set of nodes without MPI capabilities. They are virtual machines with 32GB each. They can be used for higher demand applications that would interfere with the login nodes, and therefore with other users of these login machines. A user would submit interactive jobs to individual virtual machines and avoid any significant load on login nodes.&lt;br /&gt;
&lt;br /&gt;
===The Queue Summary Script (qsum)===&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
$ qsum&lt;br /&gt;
=============== a4000 ==========================================================&lt;br /&gt;
Queue: &amp;quot;a4000&amp;quot; / nodes: 3 / down: 0 / offline: 0 / busy: 0 / available: 3&lt;br /&gt;
      AVAILABLE (3): g001, g002, g003&lt;br /&gt;
=============== a6000 ==========================================================&lt;br /&gt;
Queue: &amp;quot;a6000&amp;quot; / nodes: 1 / down: 0 / offline: 0 / busy: 0 / available: 1&lt;br /&gt;
      AVAILABLE (1): lambda01&lt;br /&gt;
=============== amd16 ==========================================================&lt;br /&gt;
Queue: &amp;quot;amd16&amp;quot; / nodes: 33 / down: 2 / offline: 0 / busy: 2 / available: 29&lt;br /&gt;
           DOWN (2): n017, n030&lt;br /&gt;
            ley (2): n001, n002&lt;br /&gt;
     AVAILABLE (29): n003, n004, n005, n006, n007, n008, n009, n010, n011, n012&lt;br /&gt;
                     n013, n014, n015, n016, n018, n019, n020, n021, n022, n023&lt;br /&gt;
                     n024, n025, n026, n027, n028, n029, n031, n032, n039&lt;br /&gt;
=============== epyc1 ==========================================================&lt;br /&gt;
Queue: &amp;quot;epyc1&amp;quot; / nodes: 1 / down: 0 / offline: 0 / busy: 0 / available: 1&lt;br /&gt;
      AVAILABLE (1): a027&lt;br /&gt;
=============== epyc2 ==========================================================&lt;br /&gt;
Queue: &amp;quot;epyc2&amp;quot; / nodes: 20 / down: 0 / offline: 0 / busy: 5 / available: 15&lt;br /&gt;
            ley (2): a030, a031&lt;br /&gt;
         msitek (3): a028, a029, a032&lt;br /&gt;
     AVAILABLE (15): a033, a034, a035, a036, a037, a038, a039, a040, a041, a042&lt;br /&gt;
                     a043, a044, a045, a046, a047&lt;br /&gt;
=============== virtual ========================================================&lt;br /&gt;
Queue: &amp;quot;virtual&amp;quot; / nodes: 6 / down: 0 / offline: 0 / busy: 0 / available: 6&lt;br /&gt;
      AVAILABLE (6): v001, v002, v003, v004, v005, v006&lt;br /&gt;
=============== xeon28 =========================================================&lt;br /&gt;
Queue: &amp;quot;xeon28&amp;quot; / nodes: 12 / down: 0 / offline: 0 / busy: 0 / available: 12&lt;br /&gt;
     AVAILABLE (12): p001, p002, p003, p004, p005, p006, p007, p008, p009, p010&lt;br /&gt;
                     p011, p012&lt;br /&gt;
================================================================================&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
==Queue Status and Monitoring Jobs==&lt;br /&gt;
&lt;br /&gt;
===qstat===&lt;br /&gt;
&lt;br /&gt;
To find out out about the status of all running jobs on the cluster you can use the &#039;&#039;&#039;&amp;quot;qstat&amp;quot;&#039;&#039;&#039; command. Here is an example:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
$ qstat&lt;br /&gt;
&lt;br /&gt;
Nov 20 18:30 ley@login3:Plots$ qstat&lt;br /&gt;
Job id            Name             User              Time Use S Queue&lt;br /&gt;
----------------  ---------------- ----------------  -------- - -----&lt;br /&gt;
3023.pbs          STDIN            msitek            4144:14* R epyc2           &lt;br /&gt;
3029.pbs          STDIN            ley               76:46:53 R epyc2           &lt;br /&gt;
3032.pbs          STDIN            msitek            2879:52* R epyc2           &lt;br /&gt;
3033.pbs          STDIN            msitek            3687:29* R epyc2           &lt;br /&gt;
3048.pbs          foo.sh           james.cook               0 Q amd16           &lt;br /&gt;
3060.pbs          of13.sh          ley               310:47:* R epyc2           &lt;br /&gt;
3061.pbs          of13.sh          ley               308:37:* R epyc2           &lt;br /&gt;
3062.pbs          of13.sh          ley               308:02:* R epyc2           &lt;br /&gt;
3063.pbs          of13.sh          ley               308:15:* R epyc2&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
The first column shows the &#039;&#039;&#039;job id&#039;&#039;&#039;, a unique identifier for all jobs ever submitted to the cluster. This job id is important when killing jobs, or for other actions you may need to take.&lt;br /&gt;
&lt;br /&gt;
The next column shows the name of the job script. If the column shows &#039;&#039;&#039;STDIN&#039;&#039;&#039;, it means that this is an &#039;&#039;&#039;interactive job&#039;&#039;&#039; where a user can enter commands in a terminal window. This is particularly useful for model and software development task where the application has to be started and killed repeatedly.&lt;br /&gt;
&lt;br /&gt;
The owner of the job is shown next. These are the user names of the various people using the cluster.&lt;br /&gt;
&lt;br /&gt;
The last three columns indicate the current run time of the job, whether it is running (R) or waiting (Q) for execution. The last entry shows the queue in which the job is running.&lt;br /&gt;
&lt;br /&gt;
===qstat -an1===&lt;br /&gt;
&lt;br /&gt;
Adding a few options gives much more detail about each jobs.&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
qstat -an1&lt;br /&gt;
&lt;br /&gt;
Nov 20 13:09 ley@login3:Plots$ qstat -an1&lt;br /&gt;
&lt;br /&gt;
pbs: &lt;br /&gt;
                                                            Req&#039;d  Req&#039;d   Elap&lt;br /&gt;
Job ID          Username Queue    Jobname    SessID NDS TSK Memory Time  S Time&lt;br /&gt;
--------------- -------- -------- ---------- ------ --- --- ------ ----- - -----&lt;br /&gt;
3023.pbs        msitek   epyc2    STDIN      24360*   1  64  350gb 100:0 R 81:46 a028/0*64&lt;br /&gt;
3029.pbs        ley      epyc2    STDIN      21719*   2 128  100gb 200:0 R 72:31 a030/0*64+a031/0*64&lt;br /&gt;
3032.pbs        msitek   epyc2    STDIN      18102*   1  64  350gb 100:0 R 57:57 a029/0*64&lt;br /&gt;
3033.pbs        msitek   epyc2    STDIN      830486   1  64  350gb 100:0 R 57:53 a032/0*64&lt;br /&gt;
3048.pbs        james.c* amd16    foo.sh        --    1  28   30gb 01:00 Q   --   -- &lt;br /&gt;
3060.pbs        ley      epyc2    STDIN      763101   1  64  350gb 48:00 R 06:42 a033/0*64&lt;br /&gt;
3061.pbs        ley      epyc2    STDIN      763947   1  64  350gb 48:00 R 06:40 a034/0*64&lt;br /&gt;
3062.pbs        ley      epyc2    STDIN      761473   1  64  350gb 48:00 R 06:39 a035/0*64&lt;br /&gt;
3063.pbs        ley      epyc2    STDIN      766205   1  64  350gb 48:00 R 06:40 a036/0*64&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
In this table, you can see how many nodes and cores are being used by each job. For example, job 3029 of the user &amp;quot;ley&amp;quot; shows that it is running on 2 nodes using a total of 128 cores. In addition to the elapsed time, the table also show the reserved time for this job. This allow you to estimate when a job will be definitely finalized (or killed by the system if still running).&lt;br /&gt;
&lt;br /&gt;
The last column (without a header) is written because the option &#039;&#039;&#039;&amp;quot;-an1&amp;quot;&#039;&#039;&#039; was used. This useful to learn about which nodes are used by each job.&lt;br /&gt;
&lt;br /&gt;
===qstat -q===&lt;br /&gt;
&lt;br /&gt;
To learn more about the queues on the cluster, the &#039;&#039;&#039;&amp;quot;-q&amp;quot;&#039;&#039;&#039; option turns out to be useful.&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
$ qstat -q&lt;br /&gt;
&lt;br /&gt;
server: pbs&lt;br /&gt;
&lt;br /&gt;
Queue            Memory CPU Time Walltime Node   Run   Que   Lm  State&lt;br /&gt;
---------------- ------ -------- -------- ---- ----- ----- ----  -----&lt;br /&gt;
virtual            30gb    --       --       1     0     0   --   E R&lt;br /&gt;
a4000             500gb    --       --       1     0     0   --   E R&lt;br /&gt;
a6000             750gb    --       --       1     0     0   --   E R&lt;br /&gt;
xeon28             --      --       --       4     0     0   --   E R&lt;br /&gt;
amd16              --      --       --       8     0     1   --   E R&lt;br /&gt;
epyc2              --      --       --       2    14     0   --   E R&lt;br /&gt;
epyc1              --      --       --       2     0     0   --   E R&lt;br /&gt;
                                               ----- -----&lt;br /&gt;
                                                  14     1&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
For each queue, some basic values are displayed. The first three queues listed above have a default memory allocation as shown, and the column &#039;&#039;&#039;&amp;quot;Node&amp;quot;&#039;&#039;&#039; indicates the maximum number of nodes that can be asked for at job submission time. For example, you can request just one node for a job from the first three queues (because these are queues where MPI makes no sense). The &#039;&#039;&#039;&amp;quot;xeon28&amp;quot;&#039;&#039;&#039; queue also for a maximum of 4 nodes per MPI job. The &#039;&#039;&#039;&amp;quot;amd16&amp;quot;&#039;&#039;&#039; queue has a maximum of 8 nodes per job, and the &#039;&#039;&#039;&amp;quot;epyc1&amp;quot;&#039;&#039;&#039; and &#039;&#039;&#039;&amp;quot;epyc2&amp;quot;&#039;&#039;&#039; queues have maxima of two nodes per job. These limitations can be changed by the administrator as needed. As shown above, this will prevent inefficient resource requests.&lt;br /&gt;
&lt;br /&gt;
===qstat -f===&lt;br /&gt;
&lt;br /&gt;
The command &#039;&#039;&#039;&amp;quot;qstat -f -F json 3029&amp;quot;&#039;&#039;&#039; retrieves extremely detailed stats on the running job 3029. The result can be returned in JSON format to be ready for further processing (shown below).&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
$ qstat -f -F json 3029&lt;br /&gt;
{&lt;br /&gt;
    &amp;quot;timestamp&amp;quot;:1763705353,&lt;br /&gt;
    &amp;quot;pbs_version&amp;quot;:&amp;quot;23.06.06&amp;quot;,&lt;br /&gt;
    &amp;quot;pbs_server&amp;quot;:&amp;quot;pbs&amp;quot;,&lt;br /&gt;
    &amp;quot;Jobs&amp;quot;:{&lt;br /&gt;
        &amp;quot;3029.pbs&amp;quot;:{&lt;br /&gt;
            &amp;quot;Job_Name&amp;quot;:&amp;quot;STDIN&amp;quot;,&lt;br /&gt;
            &amp;quot;Job_Owner&amp;quot;:&amp;quot;ley@login4&amp;quot;,&lt;br /&gt;
            &amp;quot;resources_used&amp;quot;:{&lt;br /&gt;
                &amp;quot;cpupercent&amp;quot;:98,&lt;br /&gt;
                &amp;quot;cput&amp;quot;:&amp;quot;76:46:53&amp;quot;,&lt;br /&gt;
                &amp;quot;hpmem&amp;quot;:&amp;quot;0b&amp;quot;,&lt;br /&gt;
                &amp;quot;mem&amp;quot;:&amp;quot;52428800kb&amp;quot;,&lt;br /&gt;
                &amp;quot;ncpus&amp;quot;:128,&lt;br /&gt;
                &amp;quot;vmem&amp;quot;:&amp;quot;52428800kb&amp;quot;,&lt;br /&gt;
                &amp;quot;walltime&amp;quot;:&amp;quot;78:09:32&amp;quot;&lt;br /&gt;
            },&lt;br /&gt;
            &amp;quot;job_state&amp;quot;:&amp;quot;R&amp;quot;,&lt;br /&gt;
            &amp;quot;queue&amp;quot;:&amp;quot;epyc2&amp;quot;,&lt;br /&gt;
            &amp;quot;server&amp;quot;:&amp;quot;pbs&amp;quot;,&lt;br /&gt;
            &amp;quot;Checkpoint&amp;quot;:&amp;quot;u&amp;quot;,&lt;br /&gt;
            &amp;quot;ctime&amp;quot;:&amp;quot;Mon Nov 17 17:58:25 2025&amp;quot;,&lt;br /&gt;
            &amp;quot;Error_Path&amp;quot;:&amp;quot;/dev/pts/0&amp;quot;,&lt;br /&gt;
            &amp;quot;exec_host&amp;quot;:&amp;quot;a030/0*64+a031/0*64&amp;quot;,&lt;br /&gt;
            &amp;quot;exec_vnode&amp;quot;:&amp;quot;(a030:ncpus=64:mem=52428800kb)+(a031:ncpus=64:mem=52428800kb)&amp;quot;,&lt;br /&gt;
            &amp;quot;Hold_Types&amp;quot;:&amp;quot;n&amp;quot;,&lt;br /&gt;
            &amp;quot;interactive&amp;quot;:&amp;quot;True&amp;quot;,&lt;br /&gt;
            &amp;quot;Join_Path&amp;quot;:&amp;quot;n&amp;quot;,&lt;br /&gt;
            &amp;quot;Keep_Files&amp;quot;:&amp;quot;n&amp;quot;,&lt;br /&gt;
            &amp;quot;Mail_Points&amp;quot;:&amp;quot;a&amp;quot;,&lt;br /&gt;
            &amp;quot;mtime&amp;quot;:&amp;quot;Fri Nov 21 00:07:59 2025&amp;quot;,&lt;br /&gt;
            &amp;quot;Output_Path&amp;quot;:&amp;quot;/dev/pts/0&amp;quot;,&lt;br /&gt;
            &amp;quot;Priority&amp;quot;:0,&lt;br /&gt;
            &amp;quot;qtime&amp;quot;:&amp;quot;Mon Nov 17 17:58:25 2025&amp;quot;,&lt;br /&gt;
            &amp;quot;Rerunable&amp;quot;:&amp;quot;False&amp;quot;,&lt;br /&gt;
            &amp;quot;Resource_List&amp;quot;:{&lt;br /&gt;
                &amp;quot;mem&amp;quot;:&amp;quot;100gb&amp;quot;,&lt;br /&gt;
                &amp;quot;mpiprocs&amp;quot;:128,&lt;br /&gt;
                &amp;quot;ncpus&amp;quot;:128,&lt;br /&gt;
                &amp;quot;nodect&amp;quot;:2,&lt;br /&gt;
                &amp;quot;place&amp;quot;:&amp;quot;free&amp;quot;,&lt;br /&gt;
                &amp;quot;select&amp;quot;:&amp;quot;2:ncpus=64:mem=50gb:mpiprocs=64&amp;quot;,&lt;br /&gt;
                &amp;quot;walltime&amp;quot;:&amp;quot;200:00:00&amp;quot;&lt;br /&gt;
            },&lt;br /&gt;
            &amp;quot;stime&amp;quot;:&amp;quot;Mon Nov 17 17:58:25 2025&amp;quot;,&lt;br /&gt;
            &amp;quot;session_id&amp;quot;:2171964,&lt;br /&gt;
            &amp;quot;jobdir&amp;quot;:&amp;quot;/mnt/lustre/arrow/home/ley&amp;quot;,&lt;br /&gt;
            &amp;quot;substate&amp;quot;:42,&lt;br /&gt;
            &amp;quot;Variable_List&amp;quot;:{&lt;br /&gt;
                &amp;quot;PBS_O_HOME&amp;quot;:&amp;quot;/mnt/lustre/arrow/home/ley&amp;quot;,&lt;br /&gt;
                &amp;quot;PBS_O_LANG&amp;quot;:&amp;quot;en_US.UTF-8&amp;quot;,&lt;br /&gt;
                &amp;quot;PBS_O_LOGNAME&amp;quot;:&amp;quot;ley&amp;quot;,&lt;br /&gt;
                &amp;quot;PBS_O_PATH&amp;quot;:&amp;quot;/shared/apps/active/lstc/lsprepost/SP-4.5:/shared/apps/active/lstc/lsprepost/DP-4.3:/shared/bin:/usr/share/Modules/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/pbs/bin:/opt/thinlinc/bin:/opt/thinlinc/sbin:/mnt/lustre/arrow/home/ley/.local/bin:/mnt/lustre/arrow/home/ley/bin&amp;quot;,&lt;br /&gt;
                &amp;quot;PBS_O_MAIL&amp;quot;:&amp;quot;/var/spool/mail/ley&amp;quot;,&lt;br /&gt;
                &amp;quot;PBS_O_SHELL&amp;quot;:&amp;quot;/bin/bash&amp;quot;,&lt;br /&gt;
                &amp;quot;PBS_O_WORKDIR&amp;quot;:&amp;quot;/mnt/lustre/arrow/home/ley/Qualification/LS-Dyna/Rocky9/Seatbelt/Template&amp;quot;,&lt;br /&gt;
                &amp;quot;PBS_O_SYSTEM&amp;quot;:&amp;quot;Linux&amp;quot;,&lt;br /&gt;
                &amp;quot;PBS_O_QUEUE&amp;quot;:&amp;quot;epyc2&amp;quot;,&lt;br /&gt;
                &amp;quot;PBS_O_HOST&amp;quot;:&amp;quot;login4&amp;quot;&lt;br /&gt;
            },&lt;br /&gt;
            &amp;quot;comment&amp;quot;:&amp;quot;Job run at Mon Nov 17 at 17:58 on (a030:ncpus=64:mem=52428800kb)+(a031:ncpus=64:mem=52428800kb)&amp;quot;,&lt;br /&gt;
            &amp;quot;etime&amp;quot;:&amp;quot;Mon Nov 17 17:58:25 2025&amp;quot;,&lt;br /&gt;
            &amp;quot;run_count&amp;quot;:1,&lt;br /&gt;
            &amp;quot;Submit_arguments&amp;quot;:&amp;quot;-I -q epyc2 -l walltime=200:00:00,select=2:ncpus=64:mem=50gb:mpiprocs=64&amp;quot;,&lt;br /&gt;
            &amp;quot;project&amp;quot;:&amp;quot;_pbs_project_default&amp;quot;,&lt;br /&gt;
            &amp;quot;Submit_Host&amp;quot;:&amp;quot;login4&amp;quot;&lt;br /&gt;
        }&lt;br /&gt;
    }&lt;br /&gt;
}&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
===Manual pages for qstat===&lt;br /&gt;
&lt;br /&gt;
To learn more about the &#039;&#039;&#039;&amp;quot;qstat&amp;quot;&#039;&#039;&#039; command, you can use the command &#039;&#039;&#039;&amp;quot;man qstat&amp;quot;&#039;&#039;&#039;, which will print a lot of detailed information about the capabilities of this command.&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
$ man qstat&lt;br /&gt;
&lt;br /&gt;
qstat(1B)                                       PBS Professional                                       qstat(1B)&lt;br /&gt;
&lt;br /&gt;
NAME&lt;br /&gt;
       qstat - display status of PBS jobs, queues, or servers&lt;br /&gt;
&lt;br /&gt;
SYNOPSIS&lt;br /&gt;
       Displaying Job Status&lt;br /&gt;
              Default format:&lt;br /&gt;
              qstat [-E] [-J] [-p] [-t] [-w] [-x] [[&amp;lt;job ID&amp;gt; | &amp;lt;destination&amp;gt;] ...]&lt;br /&gt;
&lt;br /&gt;
              Long format:&lt;br /&gt;
              qstat -f [-F json | dsv [-D &amp;lt;delimiter&amp;gt;]] [-E] [-J] [-p] [-t] [-w]&lt;br /&gt;
                    [-x] [[&amp;lt;job ID&amp;gt; | &amp;lt;destination&amp;gt;] ...]&lt;br /&gt;
... &amp;lt;many more pages&amp;gt;&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
==Job Submission Basics==&lt;br /&gt;
&lt;br /&gt;
Jobs are submitted into the system using the &#039;&#039;&#039;&amp;quot;qsub&amp;quot;&#039;&#039;&#039; application. This application can take many different options and allows for a lot of different resource requests to tell the cluster what to do. We are running &#039;&#039;&#039;OpenPBS 23.06.06&#039;&#039;&#039; as our job scheduler. Here is a link to the User&#039;s Manual (of PBS PRO) if you want to explore gory details and capabilities. The User&#039;s Guide has about 240 pages, the Reference Guide has 500 pages, and the Big Book has 2500 pages. So there is a lot of information available. I also added job submission info for the LCRC cluster.&lt;br /&gt;
&lt;br /&gt;
* [https://argonne-lcrc.github.io/user-guides/running-jobs-at-lcrc/pbs-pro/ Argonne&#039;s LCRC pages on job submissions on their clusters]&lt;br /&gt;
* [https://help.altair.com/2022.1.0/PBS%20Professional/PBSUserGuide2022.1.pdf PBS Professional 2022.1 User&#039;s Guide]&lt;br /&gt;
* [https://help.altair.com/2022.1.0/PBS%20Professional/PBSReferenceGuide2022.1.pdf PBS Professional 2022.1 Reference Guide]&lt;br /&gt;
* [https://help.altair.com/2022.1.0/PBS%20Professional/PBS2022.1.pdf Altair PBS Professional 2022.1 Big Book]&lt;br /&gt;
&lt;br /&gt;
The User&#039;s Guide can be very helpful to clarify some of the concepts and capabilities, but it can be hard to find the specific information you may be looking for. Please understand that we are no longer running TORQUE and MAUI, so the syntax for job submission is distinctively different yet quite similar.&lt;br /&gt;
&lt;br /&gt;
The reference guide may be helpful to understand the complete syntax and full capabilities of the software.&lt;br /&gt;
&lt;br /&gt;
The big book is what I had to use when configuring OpenPBS earlier this year. This includes all the tricky details needed to make the system work smoothly for us. It&#039;s a bit scary to look at a PDF file that is 2500 pages long, but that is nothing compared to the StarCCM+ manuals.&lt;br /&gt;
&lt;br /&gt;
&amp;lt;BLOCKQUOTE&amp;gt;&lt;br /&gt;
[[File:Attention.jpg|25px]] &#039;&#039;&#039;IMPORTANT NOTE:&#039;&#039;&#039; &#039;&#039;The following sections are important to understand. They explain how jobs are submitted and then scjeduled for execution based on resources available and the specific need of the user.&#039;&#039;&lt;br /&gt;
&amp;lt;/BLOCKQUOTE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
The following sections explain the various tasks you may want to submit fir execution, ordered from simple to complex.&lt;br /&gt;
&lt;br /&gt;
* General Batch Jobs&lt;br /&gt;
** Requesting a Single Node for a Job&lt;br /&gt;
** Requesting Multiple Nodes for a Job&lt;br /&gt;
* Embedded Job Resource Requests&lt;br /&gt;
* Interactive Jobs&lt;br /&gt;
* Interactive Jobs with X-Windows GUI Applications&lt;br /&gt;
* Running Multiple Jobs on Single Nodes&lt;br /&gt;
* Running Jobs using GPUs&lt;br /&gt;
&lt;br /&gt;
===General Batch Jobs===&lt;br /&gt;
&lt;br /&gt;
Let&#039;s get started with a very basic usage of the system. Let&#039;s assume you have a simple application, and you want to execute it on a cluster node. Let&#039;s also assume that this is a very simple application, one that runs on one or a few cores, doesn&#039;t require any keyboard interaction with the user, doesn&#039;t need the user to see what&#039;s typically written to the screen, and writes its output to files. In this case, we can submit this application as a batch job, which will place it into an execution queue and process it as soon as a node becomes available.&lt;br /&gt;
&lt;br /&gt;
If the application requires more cores than a single node can provide, we can run the application over Infiniband with MPI message passing. In this case, we need to understand the concept of MPI applications a bit better. In both cases, we get started by creating a folder on the file system. Naming conventions are important so that you can distinguish the jobs by folder name.&lt;br /&gt;
&lt;br /&gt;
For both of the above scenarios, you would typically create a Bash shell script, and then submit the script into one of the queues for eventual execution.&lt;br /&gt;
&lt;br /&gt;
====Requesting a Single Node for a Job====&lt;br /&gt;
&lt;br /&gt;
Let&#039;s try something rather trivial to get used to the concept. Create yourself a folder, for example &#039;&#039;&#039;&amp;quot;myjobfolder&amp;quot;&#039;&#039;&#039;. Within that folder, create a job submission script. That script can have any name, but something short and simple may be best. Let&#039;s assume you create a file called &#039;&#039;&#039;&amp;quot;cluster.job&amp;quot;&#039;&#039;&#039;. The file doesn&#039;t have to have that extension. Any file name will do. But using the same filename for all of your jobs helps finding your way around the many files that will be created over time. The &#039;&#039;&#039;&amp;quot;cluster.job&amp;quot;&#039;&#039;&#039; file should look something like this:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
#!/bin/bash&lt;br /&gt;
#&lt;br /&gt;
# the following ensures that you will change into the directory where you are&lt;br /&gt;
# submitting the job from.&lt;br /&gt;
cd $PBS_O_WORKDIR&lt;br /&gt;
#&lt;br /&gt;
# now we sleep for 60 seconds and waste time. This is a placeholder for your application,&lt;br /&gt;
# which would be doing useful work for you.&lt;br /&gt;
sleep 60&lt;br /&gt;
#&lt;br /&gt;
# and after doing things, we may want to write something into a file to show that&lt;br /&gt;
# our jobs is done.&lt;br /&gt;
echo `date` &amp;gt; info.log&lt;br /&gt;
#&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
This can be submitted without detailed resource specifications (except for the walltime, which is be default 0:00:00):&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
$ qsub -q virtual -l walltime=1:00:00 cluster.job &lt;br /&gt;
3072.pbs&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
Wait a little, then check the status of running jobs:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
$ qstat -an1&lt;br /&gt;
&lt;br /&gt;
pbs: &lt;br /&gt;
                                                            Req&#039;d  Req&#039;d   Elap&lt;br /&gt;
Job ID          Username Queue    Jobname    SessID NDS TSK Memory Time  S Time&lt;br /&gt;
--------------- -------- -------- ---------- ------ --- --- ------ ----- - -----&lt;br /&gt;
3023.pbs        msitek   epyc2    STDIN      24360*   1  64  350gb 100:0 R 83:17 a028/0*64&lt;br /&gt;
3029.pbs        ley      epyc2    STDIN      21719*   2 128  100gb 200:0 R 74:00 a030/0*64+a031/0*64&lt;br /&gt;
3033.pbs        msitek   epyc2    STDIN      830486   1  64  350gb 100:0 R 59:23 a032/0*64&lt;br /&gt;
3048.pbs        james.c* amd16    foo.sh        --    1  28   30gb 01:00 Q   --   -- &lt;br /&gt;
3060.pbs        ley      epyc2    STDIN      763101   1  64  350gb 48:00 R 08:10 a033/0*64&lt;br /&gt;
3061.pbs        ley      epyc2    STDIN      763947   1  64  350gb 48:00 R 08:10 a034/0*64&lt;br /&gt;
3070.pbs        ley      epyc2    STDIN      766847   1  64  350gb 48:00 R 07:23 a042/0*64&lt;br /&gt;
3072.pbs        ley      virtual  cluster.j* 230230   1   4   30gb 01:00 E 00:01 v001/0*4&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
In this particular example, we are sending this job to the &#039;&#039;&#039;queue &amp;quot;virtual&amp;quot;&#039;&#039;&#039;. This queue, by default, allocates 30GB of memory to the job, and runs on 1 node with 4 cores. This is sufficient capacity to run quite a workload. When submitting a job to a single node, &#039;&#039;&#039;reasonable maximum allocations are automatically assigned&#039;&#039;&#039;, and the user doesn&#039;t have to worry about running out of memory or how many cores he will be using.&lt;br /&gt;
&lt;br /&gt;
The only required argument is the &#039;&#039;&#039;&amp;quot;walltime&amp;quot;&#039;&#039;&#039; argument. By default, the job will quit as soon as it is submitted. This indicates to the user that he forgot to provide the &#039;&#039;&#039;&amp;quot;walltime&amp;quot;&#039;&#039;&#039; argument.&lt;br /&gt;
&lt;br /&gt;
When the job disappears from the job list, it is done. At this point, you will find the file &amp;quot;info.log&amp;quot; in your job folder.&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
$ cat info.log &lt;br /&gt;
Thu Nov 20 08:00:31 PM CST 2025&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
====Requesting Multiple Nodes for a Job====&lt;br /&gt;
&lt;br /&gt;
To run jobs on multiple nodes, you will be likely &#039;&#039;&#039;executing jobs using MPI&#039;&#039;&#039;, the message passing interface. This establishes high-speed low-latency interconnections between the cores on one machine and the cores on the other machines. Data transfer does not require involvement of the cores themselves. Instead, the core tell the InfiniBand interconnect (and cores on the same node through shared memory) to transfer the data through RDMA, remoted direct memory access. The cores don&#039;t need to spend CPU cycles on copying data, but rather simply access the data once it has been copied by the Infiniband fabric. This makes for extremely efficient remote memory access, and message passing is used to coordinate data transfer between the cores no matter where they are located on any of the nodes.&lt;br /&gt;
&lt;br /&gt;
On our cluster, MPI-aware applications like &#039;&#039;&#039;OpenFOAM&#039;&#039;&#039;, &#039;&#039;&#039;StarCCM+&#039;&#039;&#039;, and &#039;&#039;&#039;LS-Dyna&#039;&#039;&#039; can be loaded as modules, which then automatically selects the most appropriate MPI library to use. The software applications have been tested to ensure that they work out-of-the box if a user selects any specific version of any of the applications.&lt;br /&gt;
&lt;br /&gt;
The following is a very trivial example for the MPI execution of a very simple executable, with one copy running on each core of the nodes allocated to  the job. It doesn&#039;t perform any real work and just wastes resources for a short time, but it illustrates how execution on the cores of various nodes works.&lt;br /&gt;
&lt;br /&gt;
Like in the previous section, we start with a simple job script that we submit to an appropriate queues. In this case, we pick a queue that has machines with Infiniband interfaces supporting efficient communications. Let&#039;s assume we edit a file with the name &#039;&#039;&#039;&amp;quot;parallel.job&amp;quot;&#039;&#039;&#039; like this:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
#!/bin/bash&lt;br /&gt;
#&lt;br /&gt;
# the following ensures that you will change into the directory where you are&lt;br /&gt;
# submitting the job from.&lt;br /&gt;
cd $PBS_O_WORKDIR&lt;br /&gt;
#&lt;br /&gt;
# to execute a simple command on all of the cores of all of the nodes allocated to the job,&lt;br /&gt;
# we need to make one of the MPI versions available. Let&#039;s use one of the most up-to-date&lt;br /&gt;
# MPI library available on the cluster&lt;br /&gt;
module load intel/2024.2.0/mpi/2021.13&lt;br /&gt;
#&lt;br /&gt;
# now we are apply a few settings that ensure that the MPI library will use the highest-performing&lt;br /&gt;
# Infiniband Interconnect, as well as a few options to tell MPI how to interface nodes with&lt;br /&gt;
# each other and which specific Infiniband adapter to use. This is complex and requires in-depth&lt;br /&gt;
# knowledge of the QLogic Infiniband adapters we are using. It is unlikely that you will ever have to&lt;br /&gt;
# deal with these options, because the &amp;quot;module load&amp;quot; command for the engineering applications we provide&lt;br /&gt;
# on ARROW will handle all those details transparently without the user needing to understand the details.&lt;br /&gt;
export I_MPI_HYDRA_BOOTSTRAP=ssh&lt;br /&gt;
export MPI_DEVICE=rdma:ofa-v2-ib0&lt;br /&gt;
export UCX_NET_DEVICES=qib0:1&lt;br /&gt;
#&lt;br /&gt;
# it doesn&#039;t make much sense, but in this example we are executing the OS command &amp;quot;uptime&amp;quot; on all cores&lt;br /&gt;
# of the nodes allocated to this job. The output from each core is written to the file info.log. We&lt;br /&gt;
# will find 56 lines of output in the file info.log, each created by the corresponding core executing&lt;br /&gt;
# the uptime command.&lt;br /&gt;
mpirun uptime &amp;gt; info.log&lt;br /&gt;
#&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
A good queue to test scripts is the &#039;&#039;&#039;&amp;quot;xeon28&amp;quot;&#039;&#039;&#039; queue. In the queue, we have 2 14-core Xeon processers per node, so that means that each node has 56 actual cores. We do not consider hyperthreading when doing parallel computing. 56 actual cores is what&#039;s being used here. The job submission will look like this:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
qsub -q xeon28 -l walltime=1:0:0 -l select=2:ncpus=28:mpiprocs=28:mem=60G parallel.job&lt;br /&gt;
        ^                  ^               ^       ^           ^      ^ ^ ^&lt;br /&gt;
        |                  |               |       |           |      | | + --- the name of the job script to execute&lt;br /&gt;
        |                  |               |       |           |      | + ----- don&#039;t forget to specify gigabytes&lt;br /&gt;
        |                  |               |       |           |      + ------- the amount of memory to request per node&lt;br /&gt;
        |                  |               |       |           + -------------- the number of MPI tasks per nodes&lt;br /&gt;
        |                  |               |       + -------------------------- the number of cores per node&lt;br /&gt;
        |                  |               + ---------------------------------- the number of nodes to select in the queue&lt;br /&gt;
        |                   + ------------------------------------------------- the requested time, in this case 1h&lt;br /&gt;
        + --------------------------------------------------------------------- the queue to be used for the job&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
At this point, the job has created a file &amp;quot;info.log&amp;quot; with 56 lines, one per core:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
 22:26:05 up 23 days,  9:28,  0 users,  load average: 0.00, 0.00, 0.00&lt;br /&gt;
 22:26:05 up 23 days,  9:28,  0 users,  load average: 0.00, 0.00, 0.00&lt;br /&gt;
 22:26:05 up 23 days,  9:28,  0 users,  load average: 0.00, 0.00, 0.00&lt;br /&gt;
 22:26:05 up 23 days,  9:28,  0 users,  load average: 0.00, 0.00, 0.00&lt;br /&gt;
 22:26:05 up 23 days,  9:28,  0 users,  load average: 0.00, 0.00, 0.00&lt;br /&gt;
 22:26:05 up 23 days,  9:28,  0 users,  load average: 0.00, 0.00, 0.00&lt;br /&gt;
 22:26:05 up 23 days,  9:53,  0 users,  load average: 0.06, 0.03, 0.00&lt;br /&gt;
 22:26:05 up 23 days,  9:53,  0 users,  load average: 0.06, 0.03, 0.00&lt;br /&gt;
 22:26:05 up 23 days,  9:53,  0 users,  load average: 0.06, 0.03, 0.00&lt;br /&gt;
...&lt;br /&gt;
&amp;lt;/PRE&amp;gt; &lt;br /&gt;
&lt;br /&gt;
In this simple example, the lines look all the same. Upon close examination through, you can find slightly different values for some of the lines. Some lines say that the machine is up for 23 days and 9:28, while others say 23 days and 9:53. Because all 28 cores of a node would see the same uptime of the server, half of the entries show one time stamp, and the other 28 cores show the other one. That demonstrates that the 56 processes have been running independently on 2 nodes.&lt;br /&gt;
&lt;br /&gt;
===Embedded Job Resource Requests===&lt;br /&gt;
&lt;br /&gt;
The job script can be modified to embed the resource requests in form of a series of &#039;&#039;&#039;#PBS&#039;&#039;&#039; statements at the beginning of the script file. This is a very common practice use at many HPC installations and job submission engines. Let&#039;s go back to the previous example where we run the script on two nodes in parallel. That is the &#039;&#039;&#039;&amp;quot;parallel.job&amp;quot;&#039;&#039;&#039; script file again:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
#!/bin/bash&lt;br /&gt;
#&lt;br /&gt;
#PBS -q xeon28&lt;br /&gt;
#PBS -l walltime=1:0:0&lt;br /&gt;
#PBS -l select=2:ncpus=28:mpiprocs=28:mem=60G&lt;br /&gt;
#&lt;br /&gt;
# the following ensures that you will change into the directory where you are&lt;br /&gt;
# submitting the job from.&lt;br /&gt;
cd $PBS_O_WORKDIR&lt;br /&gt;
#&lt;br /&gt;
# to execute a simple command on all of the cores of all of the nodes allocated to the job,&lt;br /&gt;
# we need to make one of the MPI versions available. Let&#039;s use one of the most up-to-date&lt;br /&gt;
# MPI library available on the cluster&lt;br /&gt;
module load intel/2024.2.0/mpi/2021.13&lt;br /&gt;
#&lt;br /&gt;
# now we are apply a few settings that ensure that the MPI library will use the highest-performing&lt;br /&gt;
# Infiniband Interconnect, as well as a few options to tell MPI how to interface nodes with&lt;br /&gt;
# each other and which specific Infiniband adapter to use. This is complex and requires in-depth&lt;br /&gt;
# knowledge of the QLogic Infiniband adapters we are using. It is unlikely that you will ever have to&lt;br /&gt;
# deal with these options, because the &amp;quot;module load&amp;quot; command for the engineering applications we provide&lt;br /&gt;
# on ARROW will handle all those details transparently without the user needing to understand the details.&lt;br /&gt;
export I_MPI_HYDRA_BOOTSTRAP=ssh&lt;br /&gt;
export MPI_DEVICE=rdma:ofa-v2-ib0&lt;br /&gt;
export UCX_NET_DEVICES=qib0:1&lt;br /&gt;
#&lt;br /&gt;
# it doesn&#039;t make much sense, but in this example we are executing the OS command &amp;quot;uptime&amp;quot; on all cores&lt;br /&gt;
# of the nodes allocated to this job. The output from each core is written to the file info.log. We&lt;br /&gt;
# will find 56 lines of output in the file info.log, each created by the corresponding core executing&lt;br /&gt;
# the uptime command.&lt;br /&gt;
mpirun uptime &amp;gt; info.log&lt;br /&gt;
#&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
If the resource requests are embedded within the file, they don&#039;t have to be specified on the command line any longer (the command line overrides the embedded specifications though). This may be convenient, because all the user has to do for job submission is the following:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
qsub parallel.job&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
Here is an example with more resource specifications and job settings that affect the behavior of the job:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
#!/bin/bash&lt;br /&gt;
#&lt;br /&gt;
#PBS -q xeon28&lt;br /&gt;
#PBS -l walltime=1:0:0&lt;br /&gt;
#PBS -l select=2:ncpus=28:mpiprocs=28:mem=60G&lt;br /&gt;
#PBS -A Account&lt;br /&gt;
#PBS -j oe&lt;br /&gt;
#PBS -N JobName&lt;br /&gt;
#PBS -e log.error&lt;br /&gt;
#PBS -o log.output&lt;br /&gt;
#PBS -m bae&lt;br /&gt;
#&lt;br /&gt;
...&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
I leave this to you as an exercise to figure out what the various options mean and how to specify them. There are many more, all documented in the PBS PRO manual (see above). Most of them are not terribly relevant and can be omitted.&lt;br /&gt;
&lt;br /&gt;
===Interactive Jobs===&lt;br /&gt;
&lt;br /&gt;
On ARROW, we don&#039;t restrict queues to be used only in batch mode. While &#039;&#039;&#039;batch mode&#039;&#039;&#039; is efficient for lining up a lot of work to be executed one after the other, ARROW has been designed to &#039;&#039;&#039;allow efficient model and software development in interactive mode&#039;&#039;&#039;. We have always ensured to have more computers than minimally needed to make it possible to dedicate resources to developers as needed, even if that means wasted CPU cycles. At times, we may ask you to limit the number of interactive jobs so that a large batch workload can be processed efficiently. This happens from time to time, and we have our users coordinate this with each other.&lt;br /&gt;
&lt;br /&gt;
Let&#039;s assume that you are developing an MPI application, or you are working on a complex &#039;&#039;&#039;OpenFOAM&#039;&#039;&#039; model that requires to start parallel processes over and over again just to find a bug and then fix it quickly. To do that, you can &#039;&#039;&#039;request an interactive job&#039;&#039;&#039; by adding the &#039;&#039;&#039;&amp;quot;-I&amp;quot;&#039;&#039;&#039; option to the job submission command (this is an uppercase I). Let&#039;s go to the parallel multi-node example from above:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
qsub -I -q xeon28 -l walltime=1:0:0 -l select=2:ncpus=28:mpiprocs=28:mem=60G&lt;br /&gt;
      ^    ^                  ^               ^       ^           ^      ^ ^&lt;br /&gt;
      |    |                  |               |       |           |      | + --- don&#039;t forget to specify gigabytes&lt;br /&gt;
      |    |                  |               |       |           |      + ----- the amount of memory to request per node&lt;br /&gt;
      |    |                  |               |       |           + ------------ the number of MPI tasks per nodes&lt;br /&gt;
      |    |                  |               |       + ------------------------ the number of cores per node&lt;br /&gt;
      |    |                  |               + -------------------------------- the number of nodes to select in the queue&lt;br /&gt;
      |    |                   + ----------------------------------------------- the requested time, in this case 1h&lt;br /&gt;
      |    + ------------------------------------------------------------------- the queue to be used for the job&lt;br /&gt;
      + ------------------------------------------------------------------------ request an interactive job &amp;lt;&amp;lt;===&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
When running interactive jobs with the &#039;&#039;&#039;&amp;quot;-I&amp;quot;&#039;&#039;&#039; parameter, we don&#039;t specify av job script at the end of the submission command. The interactive job will instead start (once the nodes are available) in interactive mode, meaning that the terminal session changes over from being a series of commands executed on the login server to being a series of commands being executed on the first node of the group of nodes that are allocated to the job. At this point, you can change to the desired working directory, but what you do with the allocated resources is entirely up to you. You can load modules, including MPI libraries, and then issue the commands for your application interactively and see how they execute. If you start an &#039;&#039;&#039;&amp;quot;mpirun&amp;quot;&#039;&#039;&#039;, the cores on your allocated secondary node will work as expected. There is no difference to batch mode, other than you having the ability to execute lines of commands at will.&lt;br /&gt;
&lt;br /&gt;
===Interactive Jobs with X-Windows GUI Applications===&lt;br /&gt;
&lt;br /&gt;
Interactive use can go further than that. With some of our software applications, like &#039;&#039;&#039;StarCCM+&#039;&#039;&#039;, you can run an &#039;&#039;&#039;interactive GUI application&#039;&#039;&#039; where you control the computational work from within the applications&#039; GUI. Within the GUI, you can control execution of the numerical solver that runs on as many cores as you requested, while being able to reconfigure the case through the GUI as well. Furthermore, you can visualize developing results on the fly by creating complex plots and visualizations.&lt;br /&gt;
&lt;br /&gt;
All that is need is an option &#039;&#039;&#039;&amp;quot;-X&amp;quot;&#039;&#039;&#039; being used as part of the job submission, like this:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
qsub -X -I -q xeon28 -l walltime=1:0:0 -l select=2:ncpus=28:mpiprocs=28:mem=60G&lt;br /&gt;
      ^  ^    ^                  ^               ^       ^           ^      ^ ^&lt;br /&gt;
      |  |    |                  |               |       |           |      | + --- don&#039;t forget to specify gigabytes&lt;br /&gt;
      |  |    |                  |               |       |           |      + ----- the amount of memory to request per node&lt;br /&gt;
      |  |    |                  |               |       |           + ------------ the number of MPI tasks per nodes&lt;br /&gt;
      |  |    |                  |               |       + ------------------------ the number of cores per node&lt;br /&gt;
      |  |    |                  |               + -------------------------------- the number of nodes to select in the queue&lt;br /&gt;
      |  |    |                   + ----------------------------------------------- the requested time, in this case 1h&lt;br /&gt;
      |  |    + ------------------------------------------------------------------- the queue to be used for the job&lt;br /&gt;
      |  + ------------------------------------------------------------------------ request an interactive job&lt;br /&gt;
      + --------------------------------------------------------------------------- request GUI capabilities &amp;lt;&amp;lt;===&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
===Running Multiple Jobs on Single Nodes===&lt;br /&gt;
&lt;br /&gt;
A feature that is new on ARROW is the ability to run multiple jobs on a single node. Let&#039;s assume that you are performing a sensitivity analysis on an existing model, and the model is simple enough to return results within a reasonable time on just a few cores of a higher end machine (maybe you are running SMP versions of &#039;&#039;&#039;LS-Dyna&#039;&#039;&#039;). Our high end machines have 64 cores, so lets assume you have an &#039;&#039;&#039;LS-Dyna&#039;&#039;&#039; model that runs well on 8 cores and doesn&#039;t use a whole lot of memory. In this case, you can submit individual jobs that request simply 8 cores and a fraction of the available memory available on the node, and all jobs execute independently from each other. Each job is fit into a slot where available. It is not very different from using whole nodes for everything. The important consideration is that each job is cleanly constrained into it&#039;s allotted resources using the &#039;&#039;&#039;CGROUPS&#039;&#039;&#039; functionality of modern operating systems. Because an abusive user cannot use more cores or more memory than allocated to his job, other users can safely run smaller jobs on the same node.&lt;br /&gt;
&lt;br /&gt;
Lets assume that we have a number of smaller jobs that we want to run on a single node in the &#039;&#039;&#039;&amp;quot;xeon28&amp;quot;&#039;&#039;&#039; queue. Each job would be submitted by using reduced resources that allow for sharing but that  guarantee that the jobs will be run successfully. In this case, you can &#039;&#039;&#039;submit many jobs&#039;&#039;&#039; in the following manner (with a job script for the small jobs, each of which can &#039;&#039;&#039;request varying resources&#039;&#039;&#039; if needed - some may want to run on 5 cores, others on 3):&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
#!/bin/bash&lt;br /&gt;
#&lt;br /&gt;
# the following ensures that you will change into the directory where you are&lt;br /&gt;
# submitting the job from.&lt;br /&gt;
cd $PBS_O_WORKDIR&lt;br /&gt;
#&lt;br /&gt;
# now we sleep for 300 seconds and waste time. This is a placeholder for your application,&lt;br /&gt;
# which would be doing useful work for you.&lt;br /&gt;
sleep 300&lt;br /&gt;
#&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
Now we submit a variety of these jobs (11 total in this example) to the &#039;&#039;&#039;&amp;quot;xeon28&amp;quot;&#039;&#039;&#039; queue for execution (note that the first few jobs request different amounts of memory and core counts):&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
qsub -q xeon28 -l walltime=1:0:0 -l select=1:ncpus=12:mpiprocs=12:mem=5G small.job &lt;br /&gt;
qsub -q xeon28 -l walltime=1:0:0 -l select=1:ncpus=10:mpiprocs=10:mem=7G small.job &lt;br /&gt;
qsub -q xeon28 -l walltime=1:0:0 -l select=1:ncpus=8:mpiprocs=8:mem=9G small.job &lt;br /&gt;
qsub -q xeon28 -l walltime=1:0:0 -l select=1:ncpus=16:mpiprocs=16:mem=20G small.job &lt;br /&gt;
qsub -q xeon28 -l walltime=1:0:0 -l select=1:ncpus=2:mpiprocs=2:mem=2G small.job &lt;br /&gt;
qsub -q xeon28 -l walltime=1:0:0 -l select=1:ncpus=2:mpiprocs=2:mem=2G small.job &lt;br /&gt;
qsub -q xeon28 -l walltime=1:0:0 -l select=1:ncpus=2:mpiprocs=2:mem=2G small.job &lt;br /&gt;
qsub -q xeon28 -l walltime=1:0:0 -l select=1:ncpus=2:mpiprocs=2:mem=2G small.job &lt;br /&gt;
qsub -q xeon28 -l walltime=1:0:0 -l select=1:ncpus=2:mpiprocs=2:mem=2G small.job &lt;br /&gt;
qsub -q xeon28 -l walltime=1:0:0 -l select=1:ncpus=2:mpiprocs=2:mem=2G small.job &lt;br /&gt;
qsub -q xeon28 -l walltime=1:0:0 -l select=1:ncpus=2:mpiprocs=2:mem=2G small.job &lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
They are now running in the order of submission, allocated on as few nodes in the &amp;quot;xeon28&amp;quot; queue as necessary. Only 2 nodes are being loaded quite heavily, and 4 more cores are in use on a third node.&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
Nov 20 23:34 ley@login3:myjobfolder$ qstat -an1&lt;br /&gt;
&lt;br /&gt;
pbs: &lt;br /&gt;
                                                            Req&#039;d  Req&#039;d   Elap&lt;br /&gt;
Job ID          Username Queue    Jobname    SessID NDS TSK Memory Time  S Time&lt;br /&gt;
--------------- -------- -------- ---------- ------ --- --- ------ ----- - -----&lt;br /&gt;
&lt;br /&gt;
3082.pbs        ley      xeon28   small.job  813221   1  12    5gb 01:00 R 00:01 p001/0*12&lt;br /&gt;
3083.pbs        ley      xeon28   small.job  813288   1  10    7gb 01:00 R 00:01 p001/1*10&lt;br /&gt;
3084.pbs        ley      xeon28   small.job  671792   1   8    9gb 01:00 R 00:01 p002/0*8&lt;br /&gt;
3085.pbs        ley      xeon28   small.job  671845   1  16   20gb 01:00 R 00:01 p002/1*16&lt;br /&gt;
3086.pbs        ley      xeon28   small.job  813361   1   2    2gb 01:00 R 00:00 p001/2*2&lt;br /&gt;
3087.pbs        ley      xeon28   small.job  813413   1   2    2gb 01:00 R 00:00 p001/3*2&lt;br /&gt;
3088.pbs        ley      xeon28   small.job  813464   1   2    2gb 01:00 R 00:00 p001/4*2&lt;br /&gt;
3089.pbs        ley      xeon28   small.job  671912   1   2    2gb 01:00 R 00:00 p002/2*2&lt;br /&gt;
3090.pbs        ley      xeon28   small.job  671969   1   2    2gb 01:00 R 00:00 p002/3*2&lt;br /&gt;
3091.pbs        ley      xeon28   small.job  632092   1   2    2gb 01:00 R 00:00 p003/0*2&lt;br /&gt;
3092.pbs        ley      xeon28   small.job  632100   1   2    2gb 01:00 R 00:00 p003/1*2&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
This is a particularly effective strategy to run concurrently many cases that don&#039;t scale well beyond a few cores. When running them on fewer cores but many of them at the same time, the overall processing rate will be much higher than executing them the traditional way.&lt;br /&gt;
&lt;br /&gt;
===Running Jobs using GPUs===&lt;br /&gt;
&lt;br /&gt;
The principle of running multiple jobs on a single node becomes particularly important when using servers equipped with &#039;&#039;&#039;GPUs&#039;&#039;&#039; for &#039;&#039;&#039;ML/AI&#039;&#039;&#039; applications. The cluster doesn&#039;t have a whole lot of &#039;&#039;&#039;GPUs&#039;&#039;&#039; at this point. We have three machines with three &#039;&#039;&#039;A4000&#039;&#039;&#039; GOUs, a &#039;&#039;&#039;total of 9 A4000 GPUs&#039;&#039;&#039;. Then we have a much more powerful single machine with our &#039;&#039;&#039;four A6000 GPUs&#039;&#039;&#039;.&lt;br /&gt;
&lt;br /&gt;
Using multiple GPUs in a single application is still something where the software has to be designed with hardware in mind. GPUs have several methods of communicating with each other, e.g. very fast &#039;&#039;&#039;NVLINK&#039;&#039;&#039; between pairs of GPUs, GPUs being directly connected to one of the two CPUs in the system and thus being able to communicate faster, and GPUs that have to jump between processors when communicating, and then the whole issue of having to go possibly through PCIe bridges.&lt;br /&gt;
&lt;br /&gt;
On our system, we are providing the ability to &#039;&#039;&#039;work mostly with individual GPUs&#039;&#039;&#039;. Users can also reserve entire nodes and develop or run applications that are adapted to that hardware, including several GPUs installed on that node. One thing we do not provide is the ability of GPU to GPU communication between nodes. Thus, a job cannot request more than one GPU node at a time.&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
qsub -q a4000 -I -l walltime=1:0:0 -l select=1:ncpus=5:mem=150G:ngpus=1&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
With these specifications, three single GPU jobs can run on a single server. Each job sees only one of the reserved GPU.&lt;br /&gt;
&lt;br /&gt;
To run a massive GPU job on 64 cores with 4 &#039;&#039;&#039;A6000 GPUs&#039;&#039;&#039;, submit the job like this:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
qsub -q a6000 -I -l walltime=1:0:0 -l select=1:ncpus=64:mem=725G:ngpus=4&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
===Manual pages for qsub===&lt;br /&gt;
&lt;br /&gt;
To learn more about the &amp;quot;qsub&amp;quot; command, you can use the command &amp;quot;man qsub&amp;quot;, which will print a lot of detailed information about the capabilities of this command.&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
$ man qsub&lt;br /&gt;
&lt;br /&gt;
qsub(1B)                                               PBS Professional                                              qsub(1B)&lt;br /&gt;
&lt;br /&gt;
NAME&lt;br /&gt;
       qsub - submit a job to PBS&lt;br /&gt;
&lt;br /&gt;
SYNOPSIS&lt;br /&gt;
       qsub [-a &amp;lt;date and time&amp;gt;] [-A &amp;lt;account string&amp;gt;] [-c &amp;lt;checkpoint spec&amp;gt;]&lt;br /&gt;
            [-C &amp;lt;directive prefix&amp;gt;] [-e &amp;lt;path&amp;gt;] [-f] [-h]&lt;br /&gt;
            [-I [-G [-- &amp;lt;GUI application/script&amp;gt;]] | [-X]] [-j &amp;lt;join&amp;gt;]&lt;br /&gt;
            [-J &amp;lt;range&amp;gt; [%&amp;lt;max subjobs]] [-k &amp;lt;discard&amp;gt;] [-l &amp;lt;resource list&amp;gt;]&lt;br /&gt;
            [-m &amp;lt;mail events&amp;gt;] [-M &amp;lt;user list&amp;gt;] [-N &amp;lt;name&amp;gt;] [-o &amp;lt;path&amp;gt;]&lt;br /&gt;
            [-p &amp;lt;priority&amp;gt;] [-P &amp;lt;project&amp;gt;] [-q &amp;lt;destination&amp;gt;] [-r &amp;lt;y|n&amp;gt;]&lt;br /&gt;
            [-R &amp;lt;remove options&amp;gt;] [-S &amp;lt;path list&amp;gt;] [-u &amp;lt;user list&amp;gt;]&lt;br /&gt;
            [-v &amp;lt;variable list&amp;gt;] [-V] [-W &amp;lt;additional attributes&amp;gt;] [-z]&lt;br /&gt;
            [- | &amp;lt;script&amp;gt; | -- &amp;lt;executable&amp;gt; [&amp;lt;arguments to executable&amp;gt;]]&lt;br /&gt;
       qsub --version&lt;br /&gt;
&lt;br /&gt;
DESCRIPTION&lt;br /&gt;
       You use the qsub command to submit a batch job to PBS.  Submitting a PBS job specifies a task, requests resources, and&lt;br /&gt;
       sets job attributes.&lt;br /&gt;
... &amp;lt;many more pages&amp;gt;&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
==LS-Dyna on the ARROW Cluster==&lt;br /&gt;
&lt;br /&gt;
===Currently Available LS-Dyna Versions===&lt;br /&gt;
&lt;br /&gt;
The following is a list of &#039;&#039;&#039;LS-Dyna versions&#039;&#039;&#039; available on &#039;&#039;&#039;ARROW&#039;&#039;&#039; after the latest reconfiguration of the system. As per LSTC/ANSYS, &#039;&#039;&#039;versions before 14.0.0 are not necessarily fully supported any longer&#039;&#039;&#039; because they are supposedly not compatible with modern operating systems and cannot be made to work reliably. We have tested the listed older versions of LS-Dyna and they have passed basic tests. They may not behave exactly as they did on the old CentOS 7 operating system, and time will show whether they can still be used or whether you will need to update your models and use a fully supported version.&lt;br /&gt;
&lt;br /&gt;
All versions are loaded using the &#039;&#039;&#039;&amp;quot;module load&amp;quot;&#039;&#039;&#039; command. Versions can be listed with the &#039;&#039;&#039;&amp;quot;module avail ls-dyna&amp;quot;&#039;&#039;&#039; command. To load one of the modules, use the following syntax:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
module load ls-dyna/14.2.0/mpi-d8-ifort190-avx512&lt;br /&gt;
            ^       ^      ^   ^  ^        ^&lt;br /&gt;
            |       |      |   |  |        + --- specify the extended instruction set needed for execution&lt;br /&gt;
            |       |      |   |  + ------------ load the version of the compiler that was used to create this &lt;br /&gt;
            |       |      |   + --------------- load the version that supports double precision variables&lt;br /&gt;
            |       |      + ------------------- load the MPP (MPI) version of LS-Dyna&lt;br /&gt;
            |       + -------------------------- load specifically version 14.2.0&lt;br /&gt;
            + ---------------------------------- load a version of LS-Dyna&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
The version string is composed of multiple elements to indicate variants in compilers and compiler options. Use the following guideline to choose an appropriate version to load:&lt;br /&gt;
&lt;br /&gt;
* &#039;&#039;&#039;&amp;quot;1&amp;quot;&#039;&#039;&#039; or &#039;&#039;&#039;&amp;quot;mpi&amp;quot;&#039;&#039;&#039; indicates whether this is a single node version of LS-Dyna (&#039;&#039;&#039;SMP&#039;&#039;&#039;) or whether this is a multi-node MPI version (&#039;&#039;&#039;MPP&#039;&#039;&#039;). All MPI versions use the &#039;&#039;&#039;IntelMPI 2022&#039;&#039;&#039; libraries which have been tested thoroughly on ARROW. MPI versions will use the Infiniband Network of ARROW for high-speed and low-latency inter-process communication using RDMA (remote direct memory access).&lt;br /&gt;
* All LS-Dyna versions are available in either &#039;&#039;&#039;floating point&#039;&#039;&#039; or &#039;&#039;&#039;double precision variants&#039;&#039;&#039;. Floating point variants use 4 bytes to represent a value, and double precision variants use 8 bytes. There are pros and cons for choosing one over the other variant. With regards to computational efficiency, both perform nearly the same because all machines are equipped with 64-bit CPUs.&lt;br /&gt;
** &#039;&#039;&#039;&amp;quot;f4&amp;quot;&#039;&#039;&#039; floating point versions&lt;br /&gt;
*** Pros: These require significantly less memory to run. Results occupy less disk space, and can be transferred significantly faster into and out of ARROW.&lt;br /&gt;
*** Cons: The numerical resolution is limited to 7 significant digits, which is often undesirable when dealing with mathematical operations on small and large numbers at the same time.&lt;br /&gt;
** &#039;&#039;&#039;&amp;quot;r8&amp;quot;&#039;&#039;&#039; double precision versions&lt;br /&gt;
*** Pros: The numerical resolution is about twice the number of significant digits compare to &amp;quot;f4&amp;quot;, which helps when when dealing with mathematical operations on small and large numbers at the same time.&lt;br /&gt;
*** Cons: These require more memory to run. Results occupy more disk space, and it takes longer to transfer data into and out of ARROW.&lt;br /&gt;
* There are two more identifiers to choose from when it comes to the variants of the executables: the specific compiler used to create the executable and the specific processor instruction set required for running the executable.&lt;br /&gt;
** For modern versions of LS-Dyna, two compilers have been used by the developers to create LS-Dyna executables: the &#039;&#039;&#039;Intel Fortran Compiler&#039;&#039;&#039; and the &#039;&#039;&#039;AOCC (AMD Optimizing C/C++ and Fortran) compiler&#039;&#039;&#039;. Both variants of the software are supported on ARROW. This gives users the opportunity to choose an alternate variant of the same LS-Dyna version when running into bugs or crashes.&lt;br /&gt;
** The variants based on the various instruction set extensions (&#039;&#039;&#039;SSE2&#039;&#039;&#039;, &#039;&#039;&#039;AVX2&#039;&#039;&#039;, &#039;&#039;&#039;AVX512&#039;&#039;&#039;, and so on) gives users even more options when choosing an alternate LS-Dyna variant of the same version when running into bugs or crashes. These instruction sets are mostly related to performance gains on specific processors. We have not performed thorough performance tests and cannot recommend specific versions right now.&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
$ module avail ls-dyna&lt;br /&gt;
--------------------------------------------- /shared/apps/modulefiles ---------------------------------------------&lt;br /&gt;
ls-dyna/09.3.1/1-d8-ifort131           ls-dyna/12.2.1/mpi-f4-ifort160-sse2    ls-dyna/14.2.0/mpi-f4-ifort190-avx512  &lt;br /&gt;
ls-dyna/09.3.1/1-f4-ifort131           ls-dyna/12.2.2/1-d8-ifort160-sse2      ls-dyna/14.2.0/mpi-f4-ifort190-sse2    &lt;br /&gt;
ls-dyna/09.3.1/mpi-d8-ifort131-avx2    ls-dyna/12.2.2/1-f4-ifort160-sse2      ls-dyna/15.0.2/1-d8-ifort190-sse2      &lt;br /&gt;
ls-dyna/09.3.1/mpi-d8-ifort131-avx512  ls-dyna/12.2.2/mpi-d8-aocc400-avx2     ls-dyna/15.0.2/1-f4-ifort190-sse2      &lt;br /&gt;
ls-dyna/09.3.1/mpi-f4-ifort131-avx2    ls-dyna/12.2.2/mpi-d8-ifort160-avx2    ls-dyna/15.0.2/mpi-d8-aocc400-avx2     &lt;br /&gt;
ls-dyna/09.3.1/mpi-f4-ifort131-avx512  ls-dyna/12.2.2/mpi-d8-ifort160-sse2    ls-dyna/15.0.2/mpi-d8-ifort190-avx2    &lt;br /&gt;
ls-dyna/10.2.0/1-d8-ifort160           ls-dyna/12.2.2/mpi-f4-aocc400-avx2     ls-dyna/15.0.2/mpi-d8-ifort190-avx512  &lt;br /&gt;
ls-dyna/10.2.0/1-f4-ifort160           ls-dyna/12.2.2/mpi-f4-ifort160-avx2    ls-dyna/15.0.2/mpi-d8-ifort190-sse2    &lt;br /&gt;
ls-dyna/11.0.0/1-d8-ifort160           ls-dyna/12.2.2/mpi-f4-ifort160-sse2    ls-dyna/15.0.2/mpi-f4-aocc400-avx2     &lt;br /&gt;
ls-dyna/11.0.0/1-f4-ifort160           ls-dyna/13.0.0/1-d8-ifort190           ls-dyna/15.0.2/mpi-f4-ifort190-avx2    &lt;br /&gt;
ls-dyna/11.1.0/1-d8-ifort160-sse2      ls-dyna/13.0.0/1-f4-ifort190           ls-dyna/15.0.2/mpi-f4-ifort190-avx512  &lt;br /&gt;
ls-dyna/11.1.0/1-f4-ifort160-sse2      ls-dyna/13.0.0/mpi-d8-ifort190-avx2    ls-dyna/15.0.2/mpi-f4-ifort190-sse2    &lt;br /&gt;
ls-dyna/11.2.0/1-d8-ifort160           ls-dyna/13.0.0/mpi-d8-ifort190-sse2    ls-dyna/16.0.0/1-d8-aocc420-avx2       &lt;br /&gt;
ls-dyna/11.2.0/1-f4-ifort160           ls-dyna/13.0.0/mpi-f4-ifort190-avx2    ls-dyna/16.0.0/1-d8-aocc420-avx512     &lt;br /&gt;
ls-dyna/11.2.0/mpi-f4-ifort160-avx2    ls-dyna/13.0.0/mpi-f4-ifort190-sse2    ls-dyna/16.0.0/1-d8-ifort190-sse2      &lt;br /&gt;
ls-dyna/11.2.0/mpi-f4-ifort160-sse2    ls-dyna/13.1.0/mpi-d8-aocc310-avx2     ls-dyna/16.0.0/1-f4-aocc420-avx2       &lt;br /&gt;
ls-dyna/11.2.1/1-d8-ifort160           ls-dyna/13.1.0/mpi-d8-ifort190-avx2    ls-dyna/16.0.0/1-f4-aocc420-avx512     &lt;br /&gt;
ls-dyna/11.2.1/1-f4-ifort160           ls-dyna/13.1.0/mpi-d8-ifort190-sse2    ls-dyna/16.0.0/1-f4-ifort190-sse2      &lt;br /&gt;
ls-dyna/11.2.1/mpi-d8-ifort160-avx2    ls-dyna/13.1.0/mpi-f4-aocc310-avx2     ls-dyna/16.0.0/mpi-d8-aocc420-avx2     &lt;br /&gt;
ls-dyna/11.2.1/mpi-d8-ifort160-sse2    ls-dyna/13.1.0/mpi-f4-ifort190-avx2    ls-dyna/16.0.0/mpi-d8-aocc420-avx512   &lt;br /&gt;
ls-dyna/11.2.1/mpi-f4-ifort160-avx2    ls-dyna/13.1.0/mpi-f4-ifort190-sse2    ls-dyna/16.0.0/mpi-d8-ifort190-avx2    &lt;br /&gt;
ls-dyna/11.2.1/mpi-f4-ifort160-sse2    ls-dyna/13.1.1/mpi-d8-ifort190-avx2    ls-dyna/16.0.0/mpi-d8-ifort190-avx512  &lt;br /&gt;
ls-dyna/11.2.2/mpi-d8-ifort160-avx2    ls-dyna/13.1.1/mpi-d8-ifort190-sse2    ls-dyna/16.0.0/mpi-d8-ifort190-sse2    &lt;br /&gt;
ls-dyna/11.2.2/mpi-d8-ifort160-sse2    ls-dyna/13.1.1/mpi-f4-ifort190-avx2    ls-dyna/16.0.0/mpi-f4-aocc420-avx2     &lt;br /&gt;
ls-dyna/11.2.2/mpi-f4-ifort160-avx2    ls-dyna/13.1.1/mpi-f4-ifort190-sse2    ls-dyna/16.0.0/mpi-f4-aocc420-avx512   &lt;br /&gt;
ls-dyna/11.2.2/mpi-f4-ifort160-sse2    ls-dyna/14.0.0/1-d8-ifort190           ls-dyna/16.0.0/mpi-f4-ifort190-avx2    &lt;br /&gt;
ls-dyna/12.1.0/1-d8-ifort160           ls-dyna/14.0.0/1-f4-ifort190           ls-dyna/16.0.0/mpi-f4-ifort190-avx512  &lt;br /&gt;
ls-dyna/12.1.0/1-f4-aocc310            ls-dyna/14.0.0/mpi-d8-aocc310-avx2     ls-dyna/16.0.0/mpi-f4-ifort190-sse2    &lt;br /&gt;
ls-dyna/12.1.0/1-f4-ifort160           ls-dyna/14.0.0/mpi-d8-ifort190-avx2    ls-dyna/16.1.0/mpi-d8-aocc420-avx2     &lt;br /&gt;
ls-dyna/12.1.0/mpi-d8-aocc310-avx2     ls-dyna/14.0.0/mpi-d8-ifort190-sse2    ls-dyna/16.1.0/mpi-d8-aocc420-avx512   &lt;br /&gt;
ls-dyna/12.1.0/mpi-d8-ifort160-avx2    ls-dyna/14.0.0/mpi-f4-ifort190-avx2    ls-dyna/16.1.0/mpi-d8-ifort190-avx2    &lt;br /&gt;
ls-dyna/12.1.0/mpi-d8-ifort160-sse2    ls-dyna/14.0.0/mpi-f4-ifort190-sse2    ls-dyna/16.1.0/mpi-d8-ifort190-avx512  &lt;br /&gt;
ls-dyna/12.1.0/mpi-f4-aocc310-avx2     ls-dyna/14.1.0/1-d8-ifort190-sse2      ls-dyna/16.1.0/mpi-d8-ifort190-sse2    &lt;br /&gt;
ls-dyna/12.1.0/mpi-f4-ifort160-avx2    ls-dyna/14.1.0/1-f4-ifort190-sse2      ls-dyna/16.1.0/mpi-f4-aocc420-avx2     &lt;br /&gt;
ls-dyna/12.1.0/mpi-f4-ifort160-sse2    ls-dyna/14.1.0/mpi-d8-aocc400-avx2     ls-dyna/16.1.0/mpi-f4-aocc420-avx512&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
===Submitting an LS-Dyna Job===&lt;br /&gt;
&lt;br /&gt;
&amp;lt;BLOCKQUOTE&amp;gt;&lt;br /&gt;
[[File:Attention.jpg|25px]] &#039;&#039;&#039;IMPORTANT NOTE:&#039;&#039;&#039; &#039;&#039;The job/queue manager can track the number of LS-Dyna licenses to some degree. If all &#039;&#039;&#039;LS-Dyna users&#039;&#039;&#039; cooperate and use a script like the one shown below when submitting their jobs, the total number of concurrent &#039;&#039;&#039;LS-Dyna licenses&#039;&#039;&#039; will be tracked by the job manager correctly. That means that users can submit any number of LS-Dyna jobs, and jobs will only start when a sufficient number of licenses is available. This is managed by the &#039;&#039;&#039;&amp;quot;dynalic&amp;quot;&#039;&#039;&#039; resource at the end of the select statement. In this example, a 2-node job on 64-core nodes will need a total of &#039;&#039;&#039;&amp;quot;dynalic=128&amp;quot;&#039;&#039;&#039; licenses. This accounting breaks down when users don&#039;t use the &#039;&#039;&#039;&amp;quot;dynalic=XXX&amp;quot;&#039;&#039;&#039; statement, or when they don&#039;t calculate the number of licenses correctly. In that case, LS-Dyna jobs of all users are subject to sudden failure when LS-Dyna licenses run out. Please understand the importance of this specific setting in your job.&#039;&#039;&lt;br /&gt;
&amp;lt;/BLOCKQUOTE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
Furthermore, careful consideration should be given with regards to choice of resources for an &#039;&#039;&#039;LS-Dyna job&#039;&#039;&#039;. With 64 cores available on a single node in the &#039;&#039;&#039;&amp;quot;epyc1&amp;quot;&#039;&#039;&#039; and &#039;&#039;&#039;&amp;quot;epyc2&amp;quot;&#039;&#039;&#039; queues, it may be counterproductive to run a job on two nodes instead of a single node. Users should run their jobs with different numbers of nodes and determine whether performance increases. It may well decrease when running a job on two or more nodes. The outcome of such tests will tell what the best allocation of resources will be.&lt;br /&gt;
&lt;br /&gt;
Most users use a job script like the following. All methods for job submission the the previous chapters apply as well, so there is a lot of flexibility:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
#!/bin/bash&lt;br /&gt;
#&lt;br /&gt;
#PBS -q epyc1&lt;br /&gt;
#PBS -l walltime=12:0:0&lt;br /&gt;
#PBS -l select=2:ncpus=64:mpiprocs=64:mem=225G:dynalic=128&lt;br /&gt;
#PBS -N JobName&lt;br /&gt;
#PBS -e log.error&lt;br /&gt;
#PBS -o log.output&lt;br /&gt;
#&lt;br /&gt;
cd $PBS_O_WORKDIR&lt;br /&gt;
#&lt;br /&gt;
module load ls-dyna/12.2.1/mpi-f4-ifort160-avx2&lt;br /&gt;
module load dynamore/current&lt;br /&gt;
#&lt;br /&gt;
mpirun ls-dyna i=main.k memory1=300m memory2=100m &amp;gt; dyna.log&lt;br /&gt;
#&lt;br /&gt;
# when using the Dynamore tools, you can start something like this at the end&lt;br /&gt;
DM.plotcprs.lnx -merge &amp;gt;&amp;gt; dyna.log&lt;br /&gt;
#&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
===LSTC Tools: LS-OPT and LS-PREPOST===&lt;br /&gt;
&lt;br /&gt;
For the new Rocky 9 cluster, I have not looked deeply into the ls-opt and ls-prepost packages that were installed. I noticed though that the LSTC server provided access to much newer versions of both software packages. If you would like to learn more or have a specific version in mind, I can easily download and install it for you.&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
$ module avail ls-opt&lt;br /&gt;
----------------------------------------------- /shared/apps/modulefiles ------------------------------------------------&lt;br /&gt;
ls-opt/5.1.1  ls-opt/6.0.0  ls-opt/7.0.0  ls-opt/7.0.2   ls-opt/2022R2  &lt;br /&gt;
ls-opt/5.2.1  ls-opt/6.1.0  ls-opt/7.0.1  ls-opt/2022R1  ls-opt/2023R1  &lt;br /&gt;
&lt;br /&gt;
$ module avail ls-prepost&lt;br /&gt;
----------------------------------------------- /shared/apps/modulefiles ------------------------------------------------&lt;br /&gt;
ls-prepost/4.5.10  ls-prepost/4.8.13  ls-prepost/4.8.30  ls-prepost/4.9.16  ls-prepost/4.10.7 &lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
===Dynamore Software===&lt;br /&gt;
&lt;br /&gt;
The Dynamore tools are available as a module:&lt;br /&gt;
&lt;br /&gt;
 module load dynamore/current&lt;br /&gt;
&lt;br /&gt;
We typically acquire a yearly license for the tools as we purchase licenses for LS-Dyna.&lt;br /&gt;
&lt;br /&gt;
===Vendor License File Installation===&lt;br /&gt;
&lt;br /&gt;
If you would like for us to install a vendor license for LS-Dyna models, please contact us for the required information. We can send you the general LS-Dyna license file to show the host ids for the license server. Using that information, your vendor should be able to create a vendor license file. Please send that file to us per Email or by other means.&lt;br /&gt;
&lt;br /&gt;
==StarCCM+ on the ARROW Cluster==&lt;br /&gt;
&lt;br /&gt;
===Currently Available StarCCM+ Versions===&lt;br /&gt;
&lt;br /&gt;
As of late 2025, we have the following versions of &#039;&#039;&#039;StarCCM+&#039;&#039;&#039; available on the cluster:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
$ module avail starccm&lt;br /&gt;
---------------------------- /shared/apps/modulefiles ----------------------------&lt;br /&gt;
starccm/15.02.007-R8  starccm/16.06.008-R8  starccm/18.06.006-R8  &lt;br /&gt;
starccm/15.02.009-R8  starccm/17.02.007-R8  starccm/19.02.009-R8  &lt;br /&gt;
starccm/15.04.008-R8  starccm/17.02.008-R8  starccm/20.04.007-R8  &lt;br /&gt;
starccm/15.06.008-R8  starccm/17.04.007-R8  starccm/20.06.007-R8  &lt;br /&gt;
starccm/16.02.008-R8  starccm/17.06.007-R8  &lt;br /&gt;
starccm/16.04.007-R8  starccm/18.04.008-R8&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
If using a &#039;&#039;&#039;single node&#039;&#039;&#039; for StarCCM+, job submission (for an interactive job) is simple and will use appropriate default settings:&lt;br /&gt;
&lt;br /&gt;
 qsub -I -X -q epyc1 -l walltime=20:00:00&lt;br /&gt;
&lt;br /&gt;
StarCCM+ can make use of the job scheduler attributes by automatically obtaining the number of cores and other resources from OpenPBS. In this case, the default number of cores and mpi processes for StarCCM+ are both 64 when using the epyc1 queue. So you can start your StarCCM+ run with:&lt;br /&gt;
&lt;br /&gt;
 module load starccm/15.02.007-R8 (or any other version)&lt;br /&gt;
 starccm+ -bs pbs&lt;br /&gt;
&lt;br /&gt;
In this case, there is no need for StarCCM+ to be told to run the case in parallel with the selected number of cores/mpiprocs.&lt;br /&gt;
&lt;br /&gt;
This can get a bit more complex when running on multiple nodes or when requesting high memory nodes. In that case you would use job submission parameters as shown below:&lt;br /&gt;
&lt;br /&gt;
 qsub -I -X -q epyc1 -l walltime=20:00:00,select=2:ncpus=64:mpiprocs=61:mem=500GB&lt;br /&gt;
&lt;br /&gt;
Requesting nodes that can satisfy those resources, two nodes with these attributes must exist. We have multiple nodes with 512GB in the epyc1 queue, meaning that this job will run on two machines that have at least the required amount of memory installed (on each node). The job will be queued until two machines like this will be available. If no machines with these resources exist, the job will stay in the queue forever. Therefore, you have to craft the submission string carefully.&lt;br /&gt;
&lt;br /&gt;
To accommodate high memory jobs, the nodes have been assigned priorities for assignment. Low memory jobs have the highest priority and will be assigned to nodes that can accommodate the request. High memory nodes have the lowest priority, meaning that they are the last ones given out to users. This makes it more likely that a high memory job can be run soon when the cluster is moderately loaded with jobs.&lt;br /&gt;
&lt;br /&gt;
StarCCM+ will always use the Intel MPI fabric. Other MPI versions do not work, even when selected on the command line.&lt;br /&gt;
&lt;br /&gt;
==OpenFOAM on the ARROW Cluster==&lt;br /&gt;
&lt;br /&gt;
===Currently Available OpenFOAM  Versions===&lt;br /&gt;
&lt;br /&gt;
As of late 2025, we have the following versions of OpenFOAM available on the cluster:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
$ module avail openfoam&lt;br /&gt;
------------ /shared/apps/modulefiles ------------&lt;br /&gt;
openfoam/9   openfoam/13      openfoam/v2312  &lt;br /&gt;
openfoam/10  openfoam/13-amd  openfoam/v2406  &lt;br /&gt;
openfoam/11  openfoam/v2212   &lt;br /&gt;
openfoam/12  openfoam/v2306&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
Contact us if you encounter problems; there can be various reasons why OpenFOAM may have trouble on certain hardware or when compiling dynamic code. When loading OpenFOAM modules, a number of dependencies will be automatically loaded for you, and you don&#039;t have to load those yourself. For example:&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
$ module load openfoam/13&lt;br /&gt;
Loading openfoam/13&lt;br /&gt;
  Loading requirement: intel/2024.2.0/mpi/2021.13 gcc/gcc-12.1.0&lt;br /&gt;
&lt;br /&gt;
$ module list&lt;br /&gt;
Currently Loaded Modulefiles:&lt;br /&gt;
 1) intel/2024.2.0/mpi/2021.13   2) gcc/gcc-12.1.0   3) openfoam/13&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
In this case, OpenFOAM 13 loads the Intel 2024 MPI module, and loads the GCC compiler 12.1. OpenFOAM was compiled from source, and has been compiled specifically with that compiler and MPI version, so it make little sense to use other compilers or MPI libraries.&lt;br /&gt;
&lt;br /&gt;
Note: We have found a problem with running the Intel 2024 MPI library in the amd64 queue. Therefore, we have a modified module that uses the Intel 2022 library (I know -- Intel 2022 gives you the 2021 MPI libraries, but that is the way Intel distributes this software):&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
$ module load openfoam/13-amd &lt;br /&gt;
Loading mpi version 2021.7.0&lt;br /&gt;
Loading openfoam/13-amd&lt;br /&gt;
  Loading requirement: intel/2022.2.0/mpi/2021.7.0 gcc/gcc-12.1.0&lt;br /&gt;
&lt;br /&gt;
$ module list&lt;br /&gt;
Currently Loaded Modulefiles:&lt;br /&gt;
 1) intel/2022.2.0/mpi/2021.7.0   2) gcc/gcc-12.1.0   3) openfoam/13-amd&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
If you are compiling OpenFOAM yourself, the modules are of little help. You would need to select the appropriate MPI version and compiler before doing so, and then consistently load them before running your OpenFOAM executables. Within the &amp;quot;etc/bashrc&amp;quot; file in the source code tree, you want to set the MPI library to INTELMPI. As usual with self-compiled versions of OpenFOAM, you would &amp;quot;source etc/bashrc&amp;quot; to set up your personal environment to run your home-brew version of OpenFOAM. Contact us if you need to learn more about compiling OpenFOAM on the system.&lt;br /&gt;
&lt;br /&gt;
==Additional Software Applications and Libraries==&lt;br /&gt;
&lt;br /&gt;
===Loadable GCC Compiler Versions===&lt;br /&gt;
&lt;br /&gt;
The Rocky 9.6 operating system uses the GCC 11.5 compiler. That should be sufficient for most users when compiling your own applications. In case you need to use either a more up-to-date compiler, or if you need an older compiler for compatibility, we make the following versions available as loadable modules.&lt;br /&gt;
&lt;br /&gt;
&amp;lt;PRE&amp;gt;&lt;br /&gt;
$ module avail gcc&lt;br /&gt;
------------ /shared/apps/modulefiles ------------&lt;br /&gt;
gcc/gcc-4.9.4  gcc/gcc-7.5.0  gcc/gcc-10.3.0  &lt;br /&gt;
gcc/gcc-5.5.0  gcc/gcc-8.5.0  gcc/gcc-11.3.0  &lt;br /&gt;
gcc/gcc-6.5.0  gcc/gcc-9.5.0  gcc/gcc-12.1.0&lt;br /&gt;
&amp;lt;/PRE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
Additional versions can be created and made available as modules as well. If you need a specific version that is not currently available, please ask us to compiler and install it. If necessary, we may be able to provide access to other compilers, for example LLVM. We do not provide access to proprietary compilers at this time.&lt;br /&gt;
&lt;br /&gt;
===MPI Libraries and Runtimes===&lt;br /&gt;
&lt;br /&gt;
While we seem to have a variety of MPI versions and flavors available to users, the only MPI versions that allow us to run software over Infiniband are the Intel MPI libraries. Some of the installed alternatives are likely to fail, or will have a set of environment variables that have to be set. All major engineering software packages that we offer are pre-configured with specific MPI versions and settings that have been tested and/or provided by the vendors.&lt;br /&gt;
&lt;br /&gt;
Note: Some MPI libraries may seem to work. They may allow your MPI application to run. But inter-process network communication may travel through the rather slow and high-latency Ethernet fabric, making MPI applications very ineffective and are probably not worth while.&lt;br /&gt;
&lt;br /&gt;
===MatLab Runtimes===&lt;br /&gt;
&lt;br /&gt;
We can install MatLAB run time libraries as needed and have them available as loadable modules. Recently, we had a problem with MatLAB run time libraries being identified as security vulnerabilities. Contact us if you need them installed for one of your projects.&lt;br /&gt;
&lt;br /&gt;
===Anaconda and variants (miniconda etc)===&lt;br /&gt;
&lt;br /&gt;
Our current practice is to have users download and install their own versions of Anaconda and its variants in their own home directories. This allows for maximum flexibility when it comes to installable software modules, and users can maintain the installation, upgrades, and maintenance themselves. If you encounter issues, please contact us. One known side effect of Anaconda installations is a performance hit when starting your software, e.g. python scripts may take 30 seconds or more to execute. This is an artefact caused by the Lustre file system, which has been designed for large files accessible from many machines simultaneously. Performance on reading many small files has not been considered and is fairly poor. Again, contact us and we will design a solution for you as needed.&lt;/div&gt;</summary>
		<author><name>Ley</name></author>
	</entry>
	<entry>
		<id>https://wiki.anl.gov/wiki_tracc/index.php?title=Main_Page&amp;diff=2484</id>
		<title>Main Page</title>
		<link rel="alternate" type="text/html" href="https://wiki.anl.gov/wiki_tracc/index.php?title=Main_Page&amp;diff=2484"/>
		<updated>2025-12-09T06:33:33Z</updated>

		<summary type="html">&lt;p&gt;Ley: /* StarCCM+ */&lt;/p&gt;
&lt;hr /&gt;
&lt;div&gt;&amp;lt;!--&lt;br /&gt;
&lt;br /&gt;
__NOTOC__&lt;br /&gt;
{| style=&amp;quot;width:100%; background:#7CA0C4; color:white&amp;quot; border=&amp;quot;0&amp;quot; cellpadding=&amp;quot;0&amp;quot; cellspacing=&amp;quot;0&amp;quot;&lt;br /&gt;
| align=&amp;quot;left&amp;quot; width=&amp;quot;99%&amp;quot;|&lt;br /&gt;
&amp;lt;font color=&amp;quot;white&amp;quot; size=&amp;quot;+1&amp;quot;&amp;gt;&amp;lt;blockquote&amp;gt;The Transportation Research and Analysis Computing Center (TRACC)&amp;lt;/blockquote&amp;gt;&amp;lt;/font&amp;gt;&amp;lt;p&amp;gt;&lt;br /&gt;
&lt;br /&gt;
&amp;lt;font color=&amp;quot;white&amp;quot;&amp;gt;&amp;lt;blockquote&amp;gt;&lt;br /&gt;
This TRACC wiki is an EXTERNAL Argonne collaboration site to support information exchange and collaboration among TRACC users, collaborators, and partners.  The purpose of this Wiki is to: &lt;br /&gt;
* Document the setup, operation, and FAQs about TRACC supported software and hardware resources. &lt;br /&gt;
* Maintain a repository for hardware and software improvements and modifications. &lt;br /&gt;
* Share TRACC technical staff, collaborator and user generated &amp;quot;How-To&amp;quot; procedures with the transportation research and development community.&lt;br /&gt;
| style=&amp;quot;width:1%&amp;quot; | [[Image:Best.JPG]]&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
--&amp;gt;&lt;br /&gt;
=The ARROW HPC Cluster at TRACC=&lt;br /&gt;
&amp;lt;I&amp;gt;Operated by Argonne National Laboratory for the U.S. Department of Transportation&amp;lt;/I&amp;gt;&lt;br /&gt;
==Introduction==&lt;br /&gt;
The ARROW HPC cluster at TRACC is being operated for the Federal Highway Administration (FHWA) and the National Highway Traffic Safety Administration (NTHSA) at the US Department of Transportation. The site operates at Argonne National Laboratory under the Transportation Research and Analysis Computing Center (TRACC). The site is funded by specific US Department of Transportation projects and is not available to the general research community.&lt;br /&gt;
&lt;br /&gt;
TRACC has combined the original hardware from the Phoenix and Zephyr clusters into a single HPC cluster called ARROW. This consolidation allows for the efficient administration of TRACC cluster services. To mitigate the problems with load balancing, the different types of hardware nodes on the ARROW cluster are partitioned into queues with specific characteristics. When new hardware is installed to expand cluster resources, it will be made typically available via a new queue.&lt;br /&gt;
&lt;br /&gt;
ARROW is arranged such that there is a set of four login nodes that are accessed by users using the SSH protocol. The login servers are configured identically so that it doesn&#039;t matter which login server a user is connected to (the caveat is that the user should connect to the same login server where he may have created a GUI desktop session). All servers and nodes share a single parallel file system, and an individual home directory is provided for each user to store their data.&lt;br /&gt;
&lt;br /&gt;
Some common software packages provided on the cluster are commercial engineering applications such as StarCCM+, and LS-Dyna. In addition, selected open source software is available, such as OpenFOAM. Specific versions of each software are selected by the user through the &amp;quot;module&amp;quot; command, which is commonly used on HPC clusters. Connections are limited to the SSH protocol at this time, and cyber security is a major aspect of our operation. We will work with you to optimize your remote operating experience. Users will be able to create and use GUI desktops using the XFCE software. These desktops will stay active until the users decides to log out. Otherwise, GUI sessions will stay active and can be reconnected to, similar to the Windows Remote Desktop software. Thus, users can connect and disconnect from these sessions at will, and network interruptions will typically not interfere with your work flow. The PBS job submission framework allows users to use HPC resources in an efficient manner, sharing a fairly complex selection of specialty systems. This includes typical HPC nodes, as well as selected GPU systems for AI/ML applications.&lt;br /&gt;
&lt;br /&gt;
The ARROW HPC cluster consists of about 200 servers and provides a substantial computing facility and research platform for USDOT, especially in the areas of Bridge and River Hydraulics as well as Vehicle Occupant Safety analysis. It has been optimized for HPC engineering applications in Computational Fluid Dynamics and Computational Structural Mechanics. &lt;br /&gt;
&amp;lt;!--&lt;br /&gt;
:* [[introduction | &#039;&#039;&#039;Introduction to the ARROW Cluster&#039;&#039;&#039;]]&amp;lt;br &amp;gt;&lt;br /&gt;
:* [[ARROW Cluster#ARROW Queues | &#039;&#039;&#039;ARROW Computing Queues&#039;&#039;&#039;]]&amp;lt;br &amp;gt;&lt;br /&gt;
:* [[Becoming A User| &#039;&#039;&#039;Becoming A User&#039;&#039;&#039;]]&amp;lt;br &amp;gt;&lt;br /&gt;
--&amp;gt;&lt;br /&gt;
&lt;br /&gt;
==User Accounts==&lt;br /&gt;
User accounts are granted to users identified by the U.S. Department of Transportation. The current sponsors for our computing facility are the Federal Highway Administration (FHWA) and the National Highway Traffic Safety Administration (NTHSA) at the US Department of Transportation. You may want to contact [mailto:TRACC_Director@anl.gov Dr. Hubert Ley] at Argonne National Laboratory to inquire about the potential support and use of the facility for one of your transportation research projects.&lt;br /&gt;
&lt;br /&gt;
If you are working on a project that is already sponsoring the facility, have your contact person at USDOT request an account for you. We will then send you a link to a form where you can enter all the relevant contact information, and we will then create the account for you. These accounts usually start with &amp;quot;ac.&amp;quot;, for example &amp;quot;ac.username&amp;quot;, and you will need to remember that when you log in to ARROW. We will communicate all details with you once the application progresses.&lt;br /&gt;
&lt;br /&gt;
==Using the Cluster==&lt;br /&gt;
&lt;br /&gt;
The following links can be used to find more detailed information on the various aspects of the ARROW cluster&#039;s design and operation. This includes information on how to access the cluster remotely, typical applications that can be used to access the cluster, the overall layout of working queues and the job submission procedure, aspects of running specific software applications, information about software compilations, selecting software modules, using graphical applications and user interfaces, information about redundancy and backups, and much more.&lt;br /&gt;
&lt;br /&gt;
:* [[How to Connect| How to Connect]]&amp;lt;br&amp;gt;&lt;br /&gt;
:* [[Transferring Files| Transferring Files]]&amp;lt;br&amp;gt;&lt;br /&gt;
:* [[Job Submission and Monitoring| Job Submission and Monitoring]]&amp;lt;br &amp;gt;&lt;br /&gt;
&lt;br /&gt;
==Software Application Documentation==&lt;br /&gt;
&lt;br /&gt;
The following links will not work when reading these page with an external browser. They are only available when viewing the pages within the Linux desktop GUI after being logged in.&lt;br /&gt;
&lt;br /&gt;
===StarCCM+===&lt;br /&gt;
&lt;br /&gt;
* Version 20.06.007-R8&lt;br /&gt;
** [http://146.139.58.10/starccm/20.06.007-R8/STAR-CCM+20.06.007-R8/doc/en/online/STARCCMP StarCCM+ Online User&#039;s Guide (English)]&lt;br /&gt;
** [http://146.139.58.10/starccm/20.06.007-R8/STAR-CCM+20.06.007-R8/doc/zh/online/STARCCMP StarCCM+ Online User&#039;s Guide (Chinese)]&lt;br /&gt;
** [http://146.139.58.10/starccm/20.06.007-R8/STAR-CCM+20.06.007-R8/doc/api/html StarCCM+ API Documentation]&lt;br /&gt;
** [http://146.139.58.10/starccm/20.06.007-R8/STAR-CCM+20.06.007-R8/doc/client/html StarCCM+ Java API Documentation]&lt;br /&gt;
** [http://146.139.58.10/starccm/20.06.007-R8/STAR-CCM+20.06.007-R8/doc/ StarCCM+ All Manuals and Release Notes]&lt;br /&gt;
&lt;br /&gt;
* Version 20.04.007-R8&lt;br /&gt;
** [http://146.139.58.10/starccm/20.04.007-R8/STAR-CCM+20.04.007-R8/doc/en/online/STARCCMP StarCCM+ Online User&#039;s Guide (English)]&lt;br /&gt;
** [http://146.139.58.10/starccm/20.04.007-R8/STAR-CCM+20.04.007-R8/doc/zh/online/STARCCMP StarCCM+ Online User&#039;s Guide (Chinese)]&lt;br /&gt;
** [http://146.139.58.10/starccm/20.04.007-R8/STAR-CCM+20.04.007-R8/doc/api/html StarCCM+ API Documentation]&lt;br /&gt;
** [http://146.139.58.10/starccm/20.04.007-R8/STAR-CCM+20.04.007-R8/doc/client/html StarCCM+ Java API Documentation]&lt;br /&gt;
** [http://146.139.58.10/starccm/20.04.007-R8/STAR-CCM+20.04.007-R8/doc/ StarCCM+ All Manuals and Release Notes]&lt;br /&gt;
&lt;br /&gt;
* Version 19.02.009-R8&lt;br /&gt;
** [http://146.139.58.10/starccm/19.02.009-R8/STAR-CCM+19.02.009-R8/doc/en/online/STARCCMP StarCCM+ Online User&#039;s Guide (English)]&lt;br /&gt;
** [http://146.139.58.10/starccm/19.02.009-R8/STAR-CCM+19.02.009-R8/doc/api/html StarCCM+ API Documentation]&lt;br /&gt;
** [http://146.139.58.10/starccm/19.02.009-R8/STAR-CCM+19.02.009-R8/doc/client/html StarCCM+ Java API Documentation]&lt;br /&gt;
** [http://146.139.58.10/starccm/19.02.009-R8/STAR-CCM+19.02.009-R8/doc/ StarCCM+ All Manuals and Release Notes]&lt;br /&gt;
&lt;br /&gt;
* Version 18.06.006-R8&lt;br /&gt;
** [http://146.139.58.10/starccm/18.06.006-R8/STAR-CCM+18.06.006-R8/doc/en/online/STARCCMP StarCCM+ Online User&#039;s Guide (English)]&lt;br /&gt;
** [http://146.139.58.10/starccm/18.06.006-R8/STAR-CCM+18.06.006-R8/doc/api/html StarCCM+ API Documentation]&lt;br /&gt;
** [http://146.139.58.10/starccm/18.06.006-R8/STAR-CCM+18.06.006-R8/doc/client/html StarCCM+ Java API Documentation]&lt;br /&gt;
** [http://146.139.58.10/starccm/18.06.006-R8/STAR-CCM+18.06.006-R8/doc/ StarCCM+ All Manuals and Release Notes]&lt;br /&gt;
&lt;br /&gt;
* Version 18.04.008-R8&lt;br /&gt;
** [http://146.139.58.10/starccm/18.04.008-R8/STAR-CCM+18.04.008-R8/doc/en/online/STARCCMP StarCCM+ Online User&#039;s Guide (English)]&lt;br /&gt;
** [http://146.139.58.10/starccm/18.04.008-R8/STAR-CCM+18.04.008-R8/doc/api/html StarCCM+ API Documentation]&lt;br /&gt;
** [http://146.139.58.10/starccm/18.04.008-R8/STAR-CCM+18.04.008-R8/doc/client/html StarCCM+ Java API Documentation]&lt;br /&gt;
** [http://146.139.58.10/starccm/18.04.008-R8/STAR-CCM+18.04.008-R8/doc/ StarCCM+ All Manuals and Release Notes]&lt;br /&gt;
&lt;br /&gt;
* Version 17.06.007-R8&lt;br /&gt;
** [http://146.139.58.10/starccm/17.06.007-R8/STAR-CCM+17.06.007-R8/doc/en/online StarCCM+ Online User&#039;s Guide (English)]&lt;br /&gt;
** [http://146.139.58.10/starccm/17.06.007-R8/STAR-CCM+17.06.007-R8/doc/api/html StarCCM+ API Documentation]&lt;br /&gt;
** [http://146.139.58.10/starccm/17.06.007-R8/STAR-CCM+17.06.007-R8/doc/client/html StarCCM+ Java API Documentation]&lt;br /&gt;
** [http://146.139.58.10/starccm/17.06.007-R8/STAR-CCM+17.06.007-R8/doc/ StarCCM+ All Manuals and Release Notes]&lt;br /&gt;
&lt;br /&gt;
* Version 17.04.007-R8&lt;br /&gt;
** [http://146.139.58.10/starccm/17.04.007-R8/STAR-CCM+17.04.007-R8/doc/en/online StarCCM+ Online User&#039;s Guide (English)]&lt;br /&gt;
** [http://146.139.58.10/starccm/17.04.007-R8/STAR-CCM+17.04.007-R8/doc/api/html StarCCM+ API Documentation]&lt;br /&gt;
** [http://146.139.58.10/starccm/17.04.007-R8/STAR-CCM+17.04.007-R8/doc/client/html StarCCM+ Java API Documentation]&lt;br /&gt;
** [http://146.139.58.10/starccm/17.04.007-R8/STAR-CCM+17.04.007-R8/doc/ StarCCM+ All Manuals and Release Notes]&lt;br /&gt;
&lt;br /&gt;
* Version 17.02.008-R8&lt;br /&gt;
** [http://146.139.58.10/starccm/17.02.008-R8/STAR-CCM+17.02.008-R8/doc/en/online StarCCM+ Online User&#039;s Guide (English)]&lt;br /&gt;
** [http://146.139.58.10/starccm/17.02.008-R8/STAR-CCM+17.02.008-R8/doc/api/html StarCCM+ API Documentation]&lt;br /&gt;
** [http://146.139.58.10/starccm/17.02.008-R8/STAR-CCM+17.02.008-R8/doc/client/html StarCCM+ Java API Documentation]&lt;br /&gt;
** [http://146.139.58.10/starccm/17.02.008-R8/STAR-CCM+17.02.008-R8/doc/ StarCCM+ All Manuals and Release Notes]&lt;br /&gt;
&lt;br /&gt;
===LS-Dyna+===&lt;br /&gt;
&lt;br /&gt;
* LS-Dyna Version R16&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R16/LS-DYNA_Manual_Theory_R16.pdf LS-Dyna Theory Manual]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R16/LS-DYNA_Manual_Vol_I_R16.pdf LS-Dyna User&#039;s Manual Vol 1]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R16/LS-DYNA_Manual_Vol_II_R16.pdf LS-Dyna User&#039;s Manual Vol 2]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R16/LS-DYNA_Manual_Vol_III_R16.pdf LS-Dyna User&#039;s Manual Vol 3]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R16/ All Manuals and Release Notes]&lt;br /&gt;
&lt;br /&gt;
* LS-Dyna Version R15&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R15/LS-DYNA_Manual_Theory_R15.pdf LS-Dyna Theory Manual]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R15/LS-DYNA_Manual_Volume_I_R15.pdf LS-Dyna User&#039;s Manual Vol 1]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R15/LS-DYNA_Manual_Volume_II_R15.pdf LS-Dyna User&#039;s Manual Vol 2]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R15/LS-DYNA_Manual_Volume_III_R15.pdf LS-Dyna User&#039;s Manual Vol 3]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R15/ All Manuals and Release Notes]&lt;br /&gt;
&lt;br /&gt;
* LS-Dyna Version R14&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R14/LS-DYNA_Manual_Theory_R14.pdf LS-Dyna Theory Manual]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R14/LS-DYNA_Manual_Volume_I_R14.pdf LS-Dyna User&#039;s Manual Vol 1]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R14/LS-DYNA_Manual_Volume_II_R14.pdf LS-Dyna User&#039;s Manual Vol 2]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R14/LS-DYNA_Manual_Volume_III_R14.pdf LS-Dyna User&#039;s Manual Vol 3]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R14/ All Manuals and Release Notes]&lt;br /&gt;
&lt;br /&gt;
* LS-Dyna Version R13&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R13/LS-DYNA_Manual_Volume_I_R13.pdf LS-Dyna User&#039;s Manual Vol 1]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R13/LS-DYNA_Manual_Volume_II_R13.pdf LS-Dyna User&#039;s Manual Vol 2]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R13/LS-DYNA_Manual_Volume_III_R13.pdf LS-Dyna User&#039;s Manual Vol 3]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R13/ All Manuals and Release Notes]&lt;br /&gt;
&lt;br /&gt;
* LS-Dyna Version R12&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R12/LS-DYNA_Manual_Volume_I_R12.pdf LS-Dyna User&#039;s Manual Vol 1]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R12/LS-DYNA_Manual_Volume_II_R12.pdf LS-Dyna User&#039;s Manual Vol 2]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R12/LS-DYNA_Manual_Volume_III_R12.pdf LS-Dyna User&#039;s Manual Vol 3]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R12/ All Manuals and Release Notes]&lt;br /&gt;
&lt;br /&gt;
* LS-Dyna Version R11&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R11/LS-DYNA_Manual_Volume_I_R11.pdf LS-Dyna User&#039;s Manual Vol 1]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R11/LS-DYNA_Manual_Volume_II_R11.pdf LS-Dyna User&#039;s Manual Vol 2]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R11/LS-DYNA_Manual_Volume_III_R11.pdf LS-Dyna User&#039;s Manual Vol 3]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R11/ All Manuals and Release Notes]&lt;br /&gt;
&lt;br /&gt;
=The links and pages shown below are outdated!=&lt;br /&gt;
&lt;br /&gt;
&amp;lt;BLOCKQUOTE&amp;gt;&lt;br /&gt;
[[File:Attention.jpg|25px]] &#039;&#039;&#039;IMPORTANT NOTE:&#039;&#039;&#039; &#039;&#039;We are updating the wiki pages with the most recent information on services, hardware, and capabilities. The information provided below is outdated and in urgent need for a update. We recommend to disregard the information for now.&#039;&#039;&lt;br /&gt;
&amp;lt;/BLOCKQUOTE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
General Information&lt;br /&gt;
:* [[Becoming_A_User| Becoming_A_User]]&amp;lt;br &amp;gt;&lt;br /&gt;
&lt;br /&gt;
Procedures&lt;br /&gt;
:* [[Transferring Files| Transferring Files]]&amp;lt;br &amp;gt;&lt;br /&gt;
:* [[Setting Up Your Environment| Setting Up Your Environment]]&amp;lt;br &amp;gt;&lt;br /&gt;
:* [[Login Nodes Versus Compute Nodes| Login Nodes Versus Compute Nodes]]&amp;lt;br &amp;gt;&lt;br /&gt;
:* [[Files &amp;amp; Storage| Files &amp;amp; Storage]]&amp;lt;br &amp;gt;&lt;br /&gt;
:* [[Backups| Backups]]&amp;lt;br &amp;gt;&lt;br /&gt;
:* [[Compiling Your Code| Compiling Your Code]]&amp;lt;br &amp;gt;&lt;br /&gt;
:* [[Running Your Code| Running Your Code]]&amp;lt;br &amp;gt;&lt;br /&gt;
:* [[Graphical Applications| Graphical Applications]]&amp;lt;br &amp;gt;&lt;br /&gt;
&lt;br /&gt;
TRACC Cluster Software&lt;br /&gt;
:* [[TRACC Cluster Software#Supported On ARROW| Supported On ARROW]]&amp;lt;br &amp;gt;&lt;br /&gt;
:* [[TRACC Cluster Software#Under Investigation| Under Investigation]]&amp;lt;br &amp;gt;&lt;br /&gt;
&lt;br /&gt;
TRACC Policies&lt;br /&gt;
:* [[TRACC Policies#Application Restrictions| Application Restrictions]]&amp;lt;br &amp;gt;&lt;br /&gt;
:* [[TRACC Policies#Data Retention| Data Retention]]&amp;lt;br &amp;gt;&lt;br /&gt;
:* [[TRACC Policies#Email| Email]]&amp;lt;br &amp;gt;&lt;br /&gt;
:* [[TRACC Policies#Login Nodes| Login Nodes]]&amp;lt;br &amp;gt;&lt;/div&gt;</summary>
		<author><name>Ley</name></author>
	</entry>
	<entry>
		<id>https://wiki.anl.gov/wiki_tracc/index.php?title=Main_Page&amp;diff=2483</id>
		<title>Main Page</title>
		<link rel="alternate" type="text/html" href="https://wiki.anl.gov/wiki_tracc/index.php?title=Main_Page&amp;diff=2483"/>
		<updated>2025-12-09T06:31:10Z</updated>

		<summary type="html">&lt;p&gt;Ley: /* Software Application Documentation */&lt;/p&gt;
&lt;hr /&gt;
&lt;div&gt;&amp;lt;!--&lt;br /&gt;
&lt;br /&gt;
__NOTOC__&lt;br /&gt;
{| style=&amp;quot;width:100%; background:#7CA0C4; color:white&amp;quot; border=&amp;quot;0&amp;quot; cellpadding=&amp;quot;0&amp;quot; cellspacing=&amp;quot;0&amp;quot;&lt;br /&gt;
| align=&amp;quot;left&amp;quot; width=&amp;quot;99%&amp;quot;|&lt;br /&gt;
&amp;lt;font color=&amp;quot;white&amp;quot; size=&amp;quot;+1&amp;quot;&amp;gt;&amp;lt;blockquote&amp;gt;The Transportation Research and Analysis Computing Center (TRACC)&amp;lt;/blockquote&amp;gt;&amp;lt;/font&amp;gt;&amp;lt;p&amp;gt;&lt;br /&gt;
&lt;br /&gt;
&amp;lt;font color=&amp;quot;white&amp;quot;&amp;gt;&amp;lt;blockquote&amp;gt;&lt;br /&gt;
This TRACC wiki is an EXTERNAL Argonne collaboration site to support information exchange and collaboration among TRACC users, collaborators, and partners.  The purpose of this Wiki is to: &lt;br /&gt;
* Document the setup, operation, and FAQs about TRACC supported software and hardware resources. &lt;br /&gt;
* Maintain a repository for hardware and software improvements and modifications. &lt;br /&gt;
* Share TRACC technical staff, collaborator and user generated &amp;quot;How-To&amp;quot; procedures with the transportation research and development community.&lt;br /&gt;
| style=&amp;quot;width:1%&amp;quot; | [[Image:Best.JPG]]&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
--&amp;gt;&lt;br /&gt;
=The ARROW HPC Cluster at TRACC=&lt;br /&gt;
&amp;lt;I&amp;gt;Operated by Argonne National Laboratory for the U.S. Department of Transportation&amp;lt;/I&amp;gt;&lt;br /&gt;
==Introduction==&lt;br /&gt;
The ARROW HPC cluster at TRACC is being operated for the Federal Highway Administration (FHWA) and the National Highway Traffic Safety Administration (NTHSA) at the US Department of Transportation. The site operates at Argonne National Laboratory under the Transportation Research and Analysis Computing Center (TRACC). The site is funded by specific US Department of Transportation projects and is not available to the general research community.&lt;br /&gt;
&lt;br /&gt;
TRACC has combined the original hardware from the Phoenix and Zephyr clusters into a single HPC cluster called ARROW. This consolidation allows for the efficient administration of TRACC cluster services. To mitigate the problems with load balancing, the different types of hardware nodes on the ARROW cluster are partitioned into queues with specific characteristics. When new hardware is installed to expand cluster resources, it will be made typically available via a new queue.&lt;br /&gt;
&lt;br /&gt;
ARROW is arranged such that there is a set of four login nodes that are accessed by users using the SSH protocol. The login servers are configured identically so that it doesn&#039;t matter which login server a user is connected to (the caveat is that the user should connect to the same login server where he may have created a GUI desktop session). All servers and nodes share a single parallel file system, and an individual home directory is provided for each user to store their data.&lt;br /&gt;
&lt;br /&gt;
Some common software packages provided on the cluster are commercial engineering applications such as StarCCM+, and LS-Dyna. In addition, selected open source software is available, such as OpenFOAM. Specific versions of each software are selected by the user through the &amp;quot;module&amp;quot; command, which is commonly used on HPC clusters. Connections are limited to the SSH protocol at this time, and cyber security is a major aspect of our operation. We will work with you to optimize your remote operating experience. Users will be able to create and use GUI desktops using the XFCE software. These desktops will stay active until the users decides to log out. Otherwise, GUI sessions will stay active and can be reconnected to, similar to the Windows Remote Desktop software. Thus, users can connect and disconnect from these sessions at will, and network interruptions will typically not interfere with your work flow. The PBS job submission framework allows users to use HPC resources in an efficient manner, sharing a fairly complex selection of specialty systems. This includes typical HPC nodes, as well as selected GPU systems for AI/ML applications.&lt;br /&gt;
&lt;br /&gt;
The ARROW HPC cluster consists of about 200 servers and provides a substantial computing facility and research platform for USDOT, especially in the areas of Bridge and River Hydraulics as well as Vehicle Occupant Safety analysis. It has been optimized for HPC engineering applications in Computational Fluid Dynamics and Computational Structural Mechanics. &lt;br /&gt;
&amp;lt;!--&lt;br /&gt;
:* [[introduction | &#039;&#039;&#039;Introduction to the ARROW Cluster&#039;&#039;&#039;]]&amp;lt;br &amp;gt;&lt;br /&gt;
:* [[ARROW Cluster#ARROW Queues | &#039;&#039;&#039;ARROW Computing Queues&#039;&#039;&#039;]]&amp;lt;br &amp;gt;&lt;br /&gt;
:* [[Becoming A User| &#039;&#039;&#039;Becoming A User&#039;&#039;&#039;]]&amp;lt;br &amp;gt;&lt;br /&gt;
--&amp;gt;&lt;br /&gt;
&lt;br /&gt;
==User Accounts==&lt;br /&gt;
User accounts are granted to users identified by the U.S. Department of Transportation. The current sponsors for our computing facility are the Federal Highway Administration (FHWA) and the National Highway Traffic Safety Administration (NTHSA) at the US Department of Transportation. You may want to contact [mailto:TRACC_Director@anl.gov Dr. Hubert Ley] at Argonne National Laboratory to inquire about the potential support and use of the facility for one of your transportation research projects.&lt;br /&gt;
&lt;br /&gt;
If you are working on a project that is already sponsoring the facility, have your contact person at USDOT request an account for you. We will then send you a link to a form where you can enter all the relevant contact information, and we will then create the account for you. These accounts usually start with &amp;quot;ac.&amp;quot;, for example &amp;quot;ac.username&amp;quot;, and you will need to remember that when you log in to ARROW. We will communicate all details with you once the application progresses.&lt;br /&gt;
&lt;br /&gt;
==Using the Cluster==&lt;br /&gt;
&lt;br /&gt;
The following links can be used to find more detailed information on the various aspects of the ARROW cluster&#039;s design and operation. This includes information on how to access the cluster remotely, typical applications that can be used to access the cluster, the overall layout of working queues and the job submission procedure, aspects of running specific software applications, information about software compilations, selecting software modules, using graphical applications and user interfaces, information about redundancy and backups, and much more.&lt;br /&gt;
&lt;br /&gt;
:* [[How to Connect| How to Connect]]&amp;lt;br&amp;gt;&lt;br /&gt;
:* [[Transferring Files| Transferring Files]]&amp;lt;br&amp;gt;&lt;br /&gt;
:* [[Job Submission and Monitoring| Job Submission and Monitoring]]&amp;lt;br &amp;gt;&lt;br /&gt;
&lt;br /&gt;
==Software Application Documentation==&lt;br /&gt;
&lt;br /&gt;
The following links will not work when reading these page with an external browser. They are only available when viewing the pages within the Linux desktop GUI after being logged in.&lt;br /&gt;
&lt;br /&gt;
===StarCCM+===&lt;br /&gt;
&lt;br /&gt;
* Version 20.06.007-R8&lt;br /&gt;
** [http://146.139.58.10/starccm/20.06.007-R8/STAR-CCM+20.06.007-R8/doc/en/online/STARCCMP StarCCM+ Online User&#039;s Guide (English)]&lt;br /&gt;
** [http://146.139.58.10/starccm/20.06.007-R8/STAR-CCM+20.06.007-R8/doc/zh/online/STARCCMP StarCCM+ Online User&#039;s Guide (Chinese)]&lt;br /&gt;
** [http://146.139.58.10/starccm/20.06.007-R8/STAR-CCM+20.06.007-R8/doc/api/html StarCCM+ API Documentation]&lt;br /&gt;
** [http://146.139.58.10/starccm/20.06.007-R8/STAR-CCM+20.06.007-R8/doc/client/html StarCCM+ Java API Documentation]&lt;br /&gt;
** [http://146.139.58.10/starccm/20.06.007-R8/STAR-CCM+20.06.007-R8/doc/ StarCCM+ All Manuals and Release Notes]&lt;br /&gt;
&lt;br /&gt;
* Version 20.04.007-R8&lt;br /&gt;
** [http://146.139.58.10/starccm/20.04.007-R8/STAR-CCM+20.04.007-R8/doc/en/online/STARCCMP StarCCM+ Online User&#039;s Guide (English)]&lt;br /&gt;
** [http://146.139.58.10/starccm/20.04.007-R8/STAR-CCM+20.04.007-R8/doc/zh/online/STARCCMP StarCCM+ Online User&#039;s Guide (Chinese)]&lt;br /&gt;
** [http://146.139.58.10/starccm/20.04.007-R8/STAR-CCM+20.04.007-R8/doc/api/html StarCCM+ API Documentation]&lt;br /&gt;
** [http://146.139.58.10/starccm/20.04.007-R8/STAR-CCM+20.04.007-R8/doc/client/html StarCCM+ Java API Documentation]&lt;br /&gt;
** [http://146.139.58.10/starccm/20.04.007-R8/STAR-CCM+20.04.007-R8/doc/ StarCCM+ All Manuals and Release Notes]&lt;br /&gt;
&lt;br /&gt;
* Version 19.02.009-R8&lt;br /&gt;
** [http://146.139.58.10/starccm/19.02.009-R8/STAR-CCM+19.02.009-R8/doc/en/online/STARCCMP StarCCM+ Online User&#039;s Guide (English)]&lt;br /&gt;
** [http://146.139.58.10/starccm/19.02.009-R8/STAR-CCM+19.02.009-R8/doc/api/html StarCCM+ API Documentation]&lt;br /&gt;
** [http://146.139.58.10/starccm/19.02.009-R8/STAR-CCM+19.02.009-R8/doc/client/html StarCCM+ Java API Documentation]&lt;br /&gt;
** [http://146.139.58.10/starccm/19.02.009-R8/STAR-CCM+19.02.009-R8/doc/ StarCCM+ All Manuals and Release Notes]&lt;br /&gt;
&lt;br /&gt;
* Version 18.06.006-R8&lt;br /&gt;
** [http://146.139.58.10/starccm/18.06.006-R8/STAR-CCM+18.06.006-R8/doc/en/online/STARCCMP StarCCM+ Online User&#039;s Guide (English)]&lt;br /&gt;
** [http://146.139.58.10/starccm/18.06.006-R8/STAR-CCM+18.06.006-R8/doc/api/html StarCCM+ API Documentation]&lt;br /&gt;
** [http://146.139.58.10/starccm/18.06.006-R8/STAR-CCM+18.06.006-R8/doc/client/html StarCCM+ Java API Documentation]&lt;br /&gt;
** [http://146.139.58.10/starccm/18.06.006-R8/STAR-CCM+18.06.006-R8/doc/ StarCCM+ All Manuals and Release Notes]&lt;br /&gt;
&lt;br /&gt;
* Version 18.04.008-R8&lt;br /&gt;
** [http://146.139.58.10/starccm/18.04.008-R8/STAR-CCM+18.04.008-R8/doc/en/online/STARCCMP StarCCM+ Online User&#039;s Guide (English)]&lt;br /&gt;
** [http://146.139.58.10/starccm/18.04.008-R8/STAR-CCM+18.04.008-R8/doc/api/html StarCCM+ API Documentation]&lt;br /&gt;
** [http://146.139.58.10/starccm/18.04.008-R8/STAR-CCM+18.04.008-R8/doc/client/html StarCCM+ Java API Documentation]&lt;br /&gt;
** [http://146.139.58.10/starccm/18.04.008-R8/STAR-CCM+18.04.008-R8/doc/ StarCCM+ All Manuals and Release Notes]&lt;br /&gt;
&lt;br /&gt;
* Version 17.06.007-R8&lt;br /&gt;
** [http://146.139.58.10/starccm/17.06.007-R8/STAR-CCM+17.06.007-R8/doc/en/online StarCCM+ Online User&#039;s Guide (English)]&lt;br /&gt;
** [http://146.139.58.10/starccm/17.06.007-R8/STAR-CCM+17.06.007-R8/doc/api/html StarCCM+ API Documentation]&lt;br /&gt;
** [http://146.139.58.10/starccm/17.06.007-R8/STAR-CCM+17.06.007-R8/doc/client/html StarCCM+ Java API Documentation]&lt;br /&gt;
** [http://146.139.58.10/starccm/17.06.007-R8/STAR-CCM+17.06.007-R8/doc/ StarCCM+ All Manuals and Release Notes]&lt;br /&gt;
&lt;br /&gt;
* Version 17.04.007-R8&lt;br /&gt;
** [http://146.139.58.10/starccm/17.04.007-R8/STAR-CCM+17.04.007-R8/doc/en/online/STARCCMP StarCCM+ Online User&#039;s Guide (English)]&lt;br /&gt;
** [http://146.139.58.10/starccm/17.04.007-R8/STAR-CCM+17.04.007-R8/doc/zh/online/STARCCMP StarCCM+ Online User&#039;s Guide (Chinese)]&lt;br /&gt;
** [http://146.139.58.10/starccm/17.04.007-R8/STAR-CCM+17.04.007-R8/doc/api/html StarCCM+ API Documentation]&lt;br /&gt;
** [http://146.139.58.10/starccm/17.04.007-R8/STAR-CCM+17.04.007-R8/doc/client/html StarCCM+ Java API Documentation]&lt;br /&gt;
** [http://146.139.58.10/starccm/17.04.007-R8/STAR-CCM+17.04.007-R8/doc/ StarCCM+ All Manuals and Release Notes]&lt;br /&gt;
&lt;br /&gt;
* Version 17.02.008-R8&lt;br /&gt;
** [http://146.139.58.10/starccm/17.02.008-R8/STAR-CCM+17.02.008-R8/doc/en/online/STARCCMP StarCCM+ Online User&#039;s Guide (English)]&lt;br /&gt;
** [http://146.139.58.10/starccm/17.02.008-R8/STAR-CCM+17.02.008-R8/doc/zh/online/STARCCMP StarCCM+ Online User&#039;s Guide (Chinese)]&lt;br /&gt;
** [http://146.139.58.10/starccm/17.02.008-R8/STAR-CCM+17.02.008-R8/doc/api/html StarCCM+ API Documentation]&lt;br /&gt;
** [http://146.139.58.10/starccm/17.02.008-R8/STAR-CCM+17.02.008-R8/doc/client/html StarCCM+ Java API Documentation]&lt;br /&gt;
** [http://146.139.58.10/starccm/17.02.008-R8/STAR-CCM+17.02.008-R8/doc/ StarCCM+ All Manuals and Release Notes]&lt;br /&gt;
&lt;br /&gt;
* Version 17.02.007-R8&lt;br /&gt;
** [http://146.139.58.10/starccm/17.02.007-R8/STAR-CCM+17.02.007-R8/doc/en/online/STARCCMP StarCCM+ Online User&#039;s Guide (English)]&lt;br /&gt;
** [http://146.139.58.10/starccm/17.02.007-R8/STAR-CCM+17.02.007-R8/doc/zh/online/STARCCMP StarCCM+ Online User&#039;s Guide (Chinese)]&lt;br /&gt;
** [http://146.139.58.10/starccm/17.02.007-R8/STAR-CCM+17.02.007-R8/doc/api/html StarCCM+ API Documentation]&lt;br /&gt;
** [http://146.139.58.10/starccm/17.02.007-R8/STAR-CCM+17.02.007-R8/doc/client/html StarCCM+ Java API Documentation]&lt;br /&gt;
** [http://146.139.58.10/starccm/17.02.007-R8/STAR-CCM+17.02.007-R8/doc/ StarCCM+ All Manuals and Release Notes]&lt;br /&gt;
&lt;br /&gt;
===LS-Dyna+===&lt;br /&gt;
&lt;br /&gt;
* LS-Dyna Version R16&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R16/LS-DYNA_Manual_Theory_R16.pdf LS-Dyna Theory Manual]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R16/LS-DYNA_Manual_Vol_I_R16.pdf LS-Dyna User&#039;s Manual Vol 1]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R16/LS-DYNA_Manual_Vol_II_R16.pdf LS-Dyna User&#039;s Manual Vol 2]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R16/LS-DYNA_Manual_Vol_III_R16.pdf LS-Dyna User&#039;s Manual Vol 3]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R16/ All Manuals and Release Notes]&lt;br /&gt;
&lt;br /&gt;
* LS-Dyna Version R15&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R15/LS-DYNA_Manual_Theory_R15.pdf LS-Dyna Theory Manual]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R15/LS-DYNA_Manual_Volume_I_R15.pdf LS-Dyna User&#039;s Manual Vol 1]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R15/LS-DYNA_Manual_Volume_II_R15.pdf LS-Dyna User&#039;s Manual Vol 2]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R15/LS-DYNA_Manual_Volume_III_R15.pdf LS-Dyna User&#039;s Manual Vol 3]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R15/ All Manuals and Release Notes]&lt;br /&gt;
&lt;br /&gt;
* LS-Dyna Version R14&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R14/LS-DYNA_Manual_Theory_R14.pdf LS-Dyna Theory Manual]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R14/LS-DYNA_Manual_Volume_I_R14.pdf LS-Dyna User&#039;s Manual Vol 1]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R14/LS-DYNA_Manual_Volume_II_R14.pdf LS-Dyna User&#039;s Manual Vol 2]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R14/LS-DYNA_Manual_Volume_III_R14.pdf LS-Dyna User&#039;s Manual Vol 3]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R14/ All Manuals and Release Notes]&lt;br /&gt;
&lt;br /&gt;
* LS-Dyna Version R13&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R13/LS-DYNA_Manual_Volume_I_R13.pdf LS-Dyna User&#039;s Manual Vol 1]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R13/LS-DYNA_Manual_Volume_II_R13.pdf LS-Dyna User&#039;s Manual Vol 2]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R13/LS-DYNA_Manual_Volume_III_R13.pdf LS-Dyna User&#039;s Manual Vol 3]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R13/ All Manuals and Release Notes]&lt;br /&gt;
&lt;br /&gt;
* LS-Dyna Version R12&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R12/LS-DYNA_Manual_Volume_I_R12.pdf LS-Dyna User&#039;s Manual Vol 1]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R12/LS-DYNA_Manual_Volume_II_R12.pdf LS-Dyna User&#039;s Manual Vol 2]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R12/LS-DYNA_Manual_Volume_III_R12.pdf LS-Dyna User&#039;s Manual Vol 3]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R12/ All Manuals and Release Notes]&lt;br /&gt;
&lt;br /&gt;
* LS-Dyna Version R11&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R11/LS-DYNA_Manual_Volume_I_R11.pdf LS-Dyna User&#039;s Manual Vol 1]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R11/LS-DYNA_Manual_Volume_II_R11.pdf LS-Dyna User&#039;s Manual Vol 2]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R11/LS-DYNA_Manual_Volume_III_R11.pdf LS-Dyna User&#039;s Manual Vol 3]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R11/ All Manuals and Release Notes]&lt;br /&gt;
&lt;br /&gt;
=The links and pages shown below are outdated!=&lt;br /&gt;
&lt;br /&gt;
&amp;lt;BLOCKQUOTE&amp;gt;&lt;br /&gt;
[[File:Attention.jpg|25px]] &#039;&#039;&#039;IMPORTANT NOTE:&#039;&#039;&#039; &#039;&#039;We are updating the wiki pages with the most recent information on services, hardware, and capabilities. The information provided below is outdated and in urgent need for a update. We recommend to disregard the information for now.&#039;&#039;&lt;br /&gt;
&amp;lt;/BLOCKQUOTE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
General Information&lt;br /&gt;
:* [[Becoming_A_User| Becoming_A_User]]&amp;lt;br &amp;gt;&lt;br /&gt;
&lt;br /&gt;
Procedures&lt;br /&gt;
:* [[Transferring Files| Transferring Files]]&amp;lt;br &amp;gt;&lt;br /&gt;
:* [[Setting Up Your Environment| Setting Up Your Environment]]&amp;lt;br &amp;gt;&lt;br /&gt;
:* [[Login Nodes Versus Compute Nodes| Login Nodes Versus Compute Nodes]]&amp;lt;br &amp;gt;&lt;br /&gt;
:* [[Files &amp;amp; Storage| Files &amp;amp; Storage]]&amp;lt;br &amp;gt;&lt;br /&gt;
:* [[Backups| Backups]]&amp;lt;br &amp;gt;&lt;br /&gt;
:* [[Compiling Your Code| Compiling Your Code]]&amp;lt;br &amp;gt;&lt;br /&gt;
:* [[Running Your Code| Running Your Code]]&amp;lt;br &amp;gt;&lt;br /&gt;
:* [[Graphical Applications| Graphical Applications]]&amp;lt;br &amp;gt;&lt;br /&gt;
&lt;br /&gt;
TRACC Cluster Software&lt;br /&gt;
:* [[TRACC Cluster Software#Supported On ARROW| Supported On ARROW]]&amp;lt;br &amp;gt;&lt;br /&gt;
:* [[TRACC Cluster Software#Under Investigation| Under Investigation]]&amp;lt;br &amp;gt;&lt;br /&gt;
&lt;br /&gt;
TRACC Policies&lt;br /&gt;
:* [[TRACC Policies#Application Restrictions| Application Restrictions]]&amp;lt;br &amp;gt;&lt;br /&gt;
:* [[TRACC Policies#Data Retention| Data Retention]]&amp;lt;br &amp;gt;&lt;br /&gt;
:* [[TRACC Policies#Email| Email]]&amp;lt;br &amp;gt;&lt;br /&gt;
:* [[TRACC Policies#Login Nodes| Login Nodes]]&amp;lt;br &amp;gt;&lt;/div&gt;</summary>
		<author><name>Ley</name></author>
	</entry>
	<entry>
		<id>https://wiki.anl.gov/wiki_tracc/index.php?title=Main_Page&amp;diff=2482</id>
		<title>Main Page</title>
		<link rel="alternate" type="text/html" href="https://wiki.anl.gov/wiki_tracc/index.php?title=Main_Page&amp;diff=2482"/>
		<updated>2025-12-09T06:23:17Z</updated>

		<summary type="html">&lt;p&gt;Ley: /* Software Application Documentation */&lt;/p&gt;
&lt;hr /&gt;
&lt;div&gt;&amp;lt;!--&lt;br /&gt;
&lt;br /&gt;
__NOTOC__&lt;br /&gt;
{| style=&amp;quot;width:100%; background:#7CA0C4; color:white&amp;quot; border=&amp;quot;0&amp;quot; cellpadding=&amp;quot;0&amp;quot; cellspacing=&amp;quot;0&amp;quot;&lt;br /&gt;
| align=&amp;quot;left&amp;quot; width=&amp;quot;99%&amp;quot;|&lt;br /&gt;
&amp;lt;font color=&amp;quot;white&amp;quot; size=&amp;quot;+1&amp;quot;&amp;gt;&amp;lt;blockquote&amp;gt;The Transportation Research and Analysis Computing Center (TRACC)&amp;lt;/blockquote&amp;gt;&amp;lt;/font&amp;gt;&amp;lt;p&amp;gt;&lt;br /&gt;
&lt;br /&gt;
&amp;lt;font color=&amp;quot;white&amp;quot;&amp;gt;&amp;lt;blockquote&amp;gt;&lt;br /&gt;
This TRACC wiki is an EXTERNAL Argonne collaboration site to support information exchange and collaboration among TRACC users, collaborators, and partners.  The purpose of this Wiki is to: &lt;br /&gt;
* Document the setup, operation, and FAQs about TRACC supported software and hardware resources. &lt;br /&gt;
* Maintain a repository for hardware and software improvements and modifications. &lt;br /&gt;
* Share TRACC technical staff, collaborator and user generated &amp;quot;How-To&amp;quot; procedures with the transportation research and development community.&lt;br /&gt;
| style=&amp;quot;width:1%&amp;quot; | [[Image:Best.JPG]]&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
--&amp;gt;&lt;br /&gt;
=The ARROW HPC Cluster at TRACC=&lt;br /&gt;
&amp;lt;I&amp;gt;Operated by Argonne National Laboratory for the U.S. Department of Transportation&amp;lt;/I&amp;gt;&lt;br /&gt;
==Introduction==&lt;br /&gt;
The ARROW HPC cluster at TRACC is being operated for the Federal Highway Administration (FHWA) and the National Highway Traffic Safety Administration (NTHSA) at the US Department of Transportation. The site operates at Argonne National Laboratory under the Transportation Research and Analysis Computing Center (TRACC). The site is funded by specific US Department of Transportation projects and is not available to the general research community.&lt;br /&gt;
&lt;br /&gt;
TRACC has combined the original hardware from the Phoenix and Zephyr clusters into a single HPC cluster called ARROW. This consolidation allows for the efficient administration of TRACC cluster services. To mitigate the problems with load balancing, the different types of hardware nodes on the ARROW cluster are partitioned into queues with specific characteristics. When new hardware is installed to expand cluster resources, it will be made typically available via a new queue.&lt;br /&gt;
&lt;br /&gt;
ARROW is arranged such that there is a set of four login nodes that are accessed by users using the SSH protocol. The login servers are configured identically so that it doesn&#039;t matter which login server a user is connected to (the caveat is that the user should connect to the same login server where he may have created a GUI desktop session). All servers and nodes share a single parallel file system, and an individual home directory is provided for each user to store their data.&lt;br /&gt;
&lt;br /&gt;
Some common software packages provided on the cluster are commercial engineering applications such as StarCCM+, and LS-Dyna. In addition, selected open source software is available, such as OpenFOAM. Specific versions of each software are selected by the user through the &amp;quot;module&amp;quot; command, which is commonly used on HPC clusters. Connections are limited to the SSH protocol at this time, and cyber security is a major aspect of our operation. We will work with you to optimize your remote operating experience. Users will be able to create and use GUI desktops using the XFCE software. These desktops will stay active until the users decides to log out. Otherwise, GUI sessions will stay active and can be reconnected to, similar to the Windows Remote Desktop software. Thus, users can connect and disconnect from these sessions at will, and network interruptions will typically not interfere with your work flow. The PBS job submission framework allows users to use HPC resources in an efficient manner, sharing a fairly complex selection of specialty systems. This includes typical HPC nodes, as well as selected GPU systems for AI/ML applications.&lt;br /&gt;
&lt;br /&gt;
The ARROW HPC cluster consists of about 200 servers and provides a substantial computing facility and research platform for USDOT, especially in the areas of Bridge and River Hydraulics as well as Vehicle Occupant Safety analysis. It has been optimized for HPC engineering applications in Computational Fluid Dynamics and Computational Structural Mechanics. &lt;br /&gt;
&amp;lt;!--&lt;br /&gt;
:* [[introduction | &#039;&#039;&#039;Introduction to the ARROW Cluster&#039;&#039;&#039;]]&amp;lt;br &amp;gt;&lt;br /&gt;
:* [[ARROW Cluster#ARROW Queues | &#039;&#039;&#039;ARROW Computing Queues&#039;&#039;&#039;]]&amp;lt;br &amp;gt;&lt;br /&gt;
:* [[Becoming A User| &#039;&#039;&#039;Becoming A User&#039;&#039;&#039;]]&amp;lt;br &amp;gt;&lt;br /&gt;
--&amp;gt;&lt;br /&gt;
&lt;br /&gt;
==User Accounts==&lt;br /&gt;
User accounts are granted to users identified by the U.S. Department of Transportation. The current sponsors for our computing facility are the Federal Highway Administration (FHWA) and the National Highway Traffic Safety Administration (NTHSA) at the US Department of Transportation. You may want to contact [mailto:TRACC_Director@anl.gov Dr. Hubert Ley] at Argonne National Laboratory to inquire about the potential support and use of the facility for one of your transportation research projects.&lt;br /&gt;
&lt;br /&gt;
If you are working on a project that is already sponsoring the facility, have your contact person at USDOT request an account for you. We will then send you a link to a form where you can enter all the relevant contact information, and we will then create the account for you. These accounts usually start with &amp;quot;ac.&amp;quot;, for example &amp;quot;ac.username&amp;quot;, and you will need to remember that when you log in to ARROW. We will communicate all details with you once the application progresses.&lt;br /&gt;
&lt;br /&gt;
==Using the Cluster==&lt;br /&gt;
&lt;br /&gt;
The following links can be used to find more detailed information on the various aspects of the ARROW cluster&#039;s design and operation. This includes information on how to access the cluster remotely, typical applications that can be used to access the cluster, the overall layout of working queues and the job submission procedure, aspects of running specific software applications, information about software compilations, selecting software modules, using graphical applications and user interfaces, information about redundancy and backups, and much more.&lt;br /&gt;
&lt;br /&gt;
:* [[How to Connect| How to Connect]]&amp;lt;br&amp;gt;&lt;br /&gt;
:* [[Transferring Files| Transferring Files]]&amp;lt;br&amp;gt;&lt;br /&gt;
:* [[Job Submission and Monitoring| Job Submission and Monitoring]]&amp;lt;br &amp;gt;&lt;br /&gt;
&lt;br /&gt;
==Software Application Documentation==&lt;br /&gt;
&lt;br /&gt;
The following links will not work when reading these page with an external browser. They are only available when viewing the pages within the Linux desktop GUI after being logged in.&lt;br /&gt;
&lt;br /&gt;
===StarCCM+===&lt;br /&gt;
&lt;br /&gt;
* Version 20.06.007-R8&lt;br /&gt;
** [http://146.139.58.10/starccm/20.06.007-R8/STAR-CCM+20.06.007-R8/doc/en/online/STARCCMP StarCCM+ Online User&#039;s Guide (English)]&lt;br /&gt;
** [http://146.139.58.10/starccm/20.06.007-R8/STAR-CCM+20.06.007-R8/doc/zh/online/STARCCMP StarCCM+ Online User&#039;s Guide (Chinese)]&lt;br /&gt;
** [http://146.139.58.10/starccm/20.06.007-R8/STAR-CCM+20.06.007-R8/doc/api/html StarCCM+ API Documentation]&lt;br /&gt;
** [http://146.139.58.10/starccm/20.06.007-R8/STAR-CCM+20.06.007-R8/doc/client/html StarCCM+ Java API Documentation]&lt;br /&gt;
** [http://146.139.58.10/starccm/20.06.007-R8/STAR-CCM+20.06.007-R8/doc/ StarCCM+ All Manuals and Release Notes]&lt;br /&gt;
&lt;br /&gt;
* Version 20.04.007-R8&lt;br /&gt;
** [http://146.139.58.10/starccm/20.04.007-R8/STAR-CCM+20.04.007-R8/doc/en/online/STARCCMP StarCCM+ Online User&#039;s Guide (English)]&lt;br /&gt;
** [http://146.139.58.10/starccm/20.04.007-R8/STAR-CCM+20.04.007-R8/doc/zh/online/STARCCMP StarCCM+ Online User&#039;s Guide (Chinese)]&lt;br /&gt;
** [http://146.139.58.10/starccm/20.04.007-R8/STAR-CCM+20.04.007-R8/doc/api/html StarCCM+ API Documentation]&lt;br /&gt;
** [http://146.139.58.10/starccm/20.04.007-R8/STAR-CCM+20.04.007-R8/doc/client/html StarCCM+ Java API Documentation]&lt;br /&gt;
** [http://146.139.58.10/starccm/20.04.007-R8/STAR-CCM+20.04.007-R8/doc/ StarCCM+ All Manuals and Release Notes]&lt;br /&gt;
&lt;br /&gt;
* Version 19.02.009-R8&lt;br /&gt;
** [http://146.139.58.10/starccm/19.02.009-R8/STAR-CCM+19.02.009-R8/doc/en/online/STARCCMP StarCCM+ Online User&#039;s Guide (English)]&lt;br /&gt;
** [http://146.139.58.10/starccm/19.02.009-R8/STAR-CCM+19.02.009-R8/doc/api/html StarCCM+ API Documentation]&lt;br /&gt;
** [http://146.139.58.10/starccm/19.02.009-R8/STAR-CCM+19.02.009-R8/doc/client/html StarCCM+ Java API Documentation]&lt;br /&gt;
** [http://146.139.58.10/starccm/19.02.009-R8/STAR-CCM+19.02.009-R8/doc/ StarCCM+ All Manuals and Release Notes]&lt;br /&gt;
&lt;br /&gt;
* Version 18.06.006-R8&lt;br /&gt;
** [http://146.139.58.10/starccm/18.06.006-R8/STAR-CCM+18.06.006-R8/doc/en/online/STARCCMP StarCCM+ Online User&#039;s Guide (English)]&lt;br /&gt;
** [http://146.139.58.10/starccm/18.06.006-R8/STAR-CCM+18.06.006-R8/doc/zh/online/STARCCMP StarCCM+ Online User&#039;s Guide (Chinese)]&lt;br /&gt;
** [http://146.139.58.10/starccm/18.06.006-R8/STAR-CCM+18.06.006-R8/doc/api/html StarCCM+ API Documentation]&lt;br /&gt;
** [http://146.139.58.10/starccm/18.06.006-R8/STAR-CCM+18.06.006-R8/doc/client/html StarCCM+ Java API Documentation]&lt;br /&gt;
** [http://146.139.58.10/starccm/18.06.006-R8/STAR-CCM+18.06.006-R8/doc/ StarCCM+ All Manuals and Release Notes]&lt;br /&gt;
&lt;br /&gt;
* Version 18.04.008-R8&lt;br /&gt;
** [http://146.139.58.10/starccm/18.04.008-R8/STAR-CCM+18.04.008-R8/doc/en/online/STARCCMP StarCCM+ Online User&#039;s Guide (English)]&lt;br /&gt;
** [http://146.139.58.10/starccm/18.04.008-R8/STAR-CCM+18.04.008-R8/doc/zh/online/STARCCMP StarCCM+ Online User&#039;s Guide (Chinese)]&lt;br /&gt;
** [http://146.139.58.10/starccm/18.04.008-R8/STAR-CCM+18.04.008-R8/doc/api/html StarCCM+ API Documentation]&lt;br /&gt;
** [http://146.139.58.10/starccm/18.04.008-R8/STAR-CCM+18.04.008-R8/doc/client/html StarCCM+ Java API Documentation]&lt;br /&gt;
** [http://146.139.58.10/starccm/18.04.008-R8/STAR-CCM+18.04.008-R8/doc/ StarCCM+ All Manuals and Release Notes]&lt;br /&gt;
&lt;br /&gt;
* Version 17.06.007-R8&lt;br /&gt;
** [http://146.139.58.10/starccm/17.06.007-R8/STAR-CCM+17.06.007-R8/doc/en/online/STARCCMP StarCCM+ Online User&#039;s Guide (English)]&lt;br /&gt;
** [http://146.139.58.10/starccm/17.06.007-R8/STAR-CCM+17.06.007-R8/doc/zh/online/STARCCMP StarCCM+ Online User&#039;s Guide (Chinese)]&lt;br /&gt;
** [http://146.139.58.10/starccm/17.06.007-R8/STAR-CCM+17.06.007-R8/doc/api/html StarCCM+ API Documentation]&lt;br /&gt;
** [http://146.139.58.10/starccm/17.06.007-R8/STAR-CCM+17.06.007-R8/doc/client/html StarCCM+ Java API Documentation]&lt;br /&gt;
** [http://146.139.58.10/starccm/17.06.007-R8/STAR-CCM+17.06.007-R8/doc/ StarCCM+ All Manuals and Release Notes]&lt;br /&gt;
&lt;br /&gt;
* Version 17.04.007-R8&lt;br /&gt;
** [http://146.139.58.10/starccm/17.04.007-R8/STAR-CCM+17.04.007-R8/doc/en/online/STARCCMP StarCCM+ Online User&#039;s Guide (English)]&lt;br /&gt;
** [http://146.139.58.10/starccm/17.04.007-R8/STAR-CCM+17.04.007-R8/doc/zh/online/STARCCMP StarCCM+ Online User&#039;s Guide (Chinese)]&lt;br /&gt;
** [http://146.139.58.10/starccm/17.04.007-R8/STAR-CCM+17.04.007-R8/doc/api/html StarCCM+ API Documentation]&lt;br /&gt;
** [http://146.139.58.10/starccm/17.04.007-R8/STAR-CCM+17.04.007-R8/doc/client/html StarCCM+ Java API Documentation]&lt;br /&gt;
** [http://146.139.58.10/starccm/17.04.007-R8/STAR-CCM+17.04.007-R8/doc/ StarCCM+ All Manuals and Release Notes]&lt;br /&gt;
&lt;br /&gt;
* Version 17.02.008-R8&lt;br /&gt;
** [http://146.139.58.10/starccm/17.02.008-R8/STAR-CCM+17.02.008-R8/doc/en/online/STARCCMP StarCCM+ Online User&#039;s Guide (English)]&lt;br /&gt;
** [http://146.139.58.10/starccm/17.02.008-R8/STAR-CCM+17.02.008-R8/doc/zh/online/STARCCMP StarCCM+ Online User&#039;s Guide (Chinese)]&lt;br /&gt;
** [http://146.139.58.10/starccm/17.02.008-R8/STAR-CCM+17.02.008-R8/doc/api/html StarCCM+ API Documentation]&lt;br /&gt;
** [http://146.139.58.10/starccm/17.02.008-R8/STAR-CCM+17.02.008-R8/doc/client/html StarCCM+ Java API Documentation]&lt;br /&gt;
** [http://146.139.58.10/starccm/17.02.008-R8/STAR-CCM+17.02.008-R8/doc/ StarCCM+ All Manuals and Release Notes]&lt;br /&gt;
&lt;br /&gt;
* Version 17.02.007-R8&lt;br /&gt;
** [http://146.139.58.10/starccm/17.02.007-R8/STAR-CCM+17.02.007-R8/doc/en/online/STARCCMP StarCCM+ Online User&#039;s Guide (English)]&lt;br /&gt;
** [http://146.139.58.10/starccm/17.02.007-R8/STAR-CCM+17.02.007-R8/doc/zh/online/STARCCMP StarCCM+ Online User&#039;s Guide (Chinese)]&lt;br /&gt;
** [http://146.139.58.10/starccm/17.02.007-R8/STAR-CCM+17.02.007-R8/doc/api/html StarCCM+ API Documentation]&lt;br /&gt;
** [http://146.139.58.10/starccm/17.02.007-R8/STAR-CCM+17.02.007-R8/doc/client/html StarCCM+ Java API Documentation]&lt;br /&gt;
** [http://146.139.58.10/starccm/17.02.007-R8/STAR-CCM+17.02.007-R8/doc/ StarCCM+ All Manuals and Release Notes]&lt;br /&gt;
&lt;br /&gt;
===LS-Dyna+===&lt;br /&gt;
&lt;br /&gt;
* LS-Dyna Version R16&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R16/LS-DYNA_Manual_Theory_R16.pdf LS-Dyna Theory Manual]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R16/LS-DYNA_Manual_Vol_I_R16.pdf LS-Dyna User&#039;s Manual Vol 1]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R16/LS-DYNA_Manual_Vol_II_R16.pdf LS-Dyna User&#039;s Manual Vol 2]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R16/LS-DYNA_Manual_Vol_III_R16.pdf LS-Dyna User&#039;s Manual Vol 3]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R16/ All Manuals and Release Notes]&lt;br /&gt;
&lt;br /&gt;
* LS-Dyna Version R15&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R15/LS-DYNA_Manual_Theory_R15.pdf LS-Dyna Theory Manual]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R15/LS-DYNA_Manual_Volume_I_R15.pdf LS-Dyna User&#039;s Manual Vol 1]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R15/LS-DYNA_Manual_Volume_II_R15.pdf LS-Dyna User&#039;s Manual Vol 2]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R15/LS-DYNA_Manual_Volume_III_R15.pdf LS-Dyna User&#039;s Manual Vol 3]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R15/ All Manuals and Release Notes]&lt;br /&gt;
&lt;br /&gt;
* LS-Dyna Version R14&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R14/LS-DYNA_Manual_Theory_R14.pdf LS-Dyna Theory Manual]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R14/LS-DYNA_Manual_Volume_I_R14.pdf LS-Dyna User&#039;s Manual Vol 1]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R14/LS-DYNA_Manual_Volume_II_R14.pdf LS-Dyna User&#039;s Manual Vol 2]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R14/LS-DYNA_Manual_Volume_III_R14.pdf LS-Dyna User&#039;s Manual Vol 3]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R14/ All Manuals and Release Notes]&lt;br /&gt;
&lt;br /&gt;
* LS-Dyna Version R13&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R13/LS-DYNA_Manual_Volume_I_R13.pdf LS-Dyna User&#039;s Manual Vol 1]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R13/LS-DYNA_Manual_Volume_II_R13.pdf LS-Dyna User&#039;s Manual Vol 2]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R13/LS-DYNA_Manual_Volume_III_R13.pdf LS-Dyna User&#039;s Manual Vol 3]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R13/ All Manuals and Release Notes]&lt;br /&gt;
&lt;br /&gt;
* LS-Dyna Version R12&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R12/LS-DYNA_Manual_Volume_I_R12.pdf LS-Dyna User&#039;s Manual Vol 1]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R12/LS-DYNA_Manual_Volume_II_R12.pdf LS-Dyna User&#039;s Manual Vol 2]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R12/LS-DYNA_Manual_Volume_III_R12.pdf LS-Dyna User&#039;s Manual Vol 3]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R12/ All Manuals and Release Notes]&lt;br /&gt;
&lt;br /&gt;
* LS-Dyna Version R11&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R11/LS-DYNA_Manual_Volume_I_R11.pdf LS-Dyna User&#039;s Manual Vol 1]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R11/LS-DYNA_Manual_Volume_II_R11.pdf LS-Dyna User&#039;s Manual Vol 2]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R11/LS-DYNA_Manual_Volume_III_R11.pdf LS-Dyna User&#039;s Manual Vol 3]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R11/ All Manuals and Release Notes]&lt;br /&gt;
&lt;br /&gt;
=The links and pages shown below are outdated!=&lt;br /&gt;
&lt;br /&gt;
&amp;lt;BLOCKQUOTE&amp;gt;&lt;br /&gt;
[[File:Attention.jpg|25px]] &#039;&#039;&#039;IMPORTANT NOTE:&#039;&#039;&#039; &#039;&#039;We are updating the wiki pages with the most recent information on services, hardware, and capabilities. The information provided below is outdated and in urgent need for a update. We recommend to disregard the information for now.&#039;&#039;&lt;br /&gt;
&amp;lt;/BLOCKQUOTE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
General Information&lt;br /&gt;
:* [[Becoming_A_User| Becoming_A_User]]&amp;lt;br &amp;gt;&lt;br /&gt;
&lt;br /&gt;
Procedures&lt;br /&gt;
:* [[Transferring Files| Transferring Files]]&amp;lt;br &amp;gt;&lt;br /&gt;
:* [[Setting Up Your Environment| Setting Up Your Environment]]&amp;lt;br &amp;gt;&lt;br /&gt;
:* [[Login Nodes Versus Compute Nodes| Login Nodes Versus Compute Nodes]]&amp;lt;br &amp;gt;&lt;br /&gt;
:* [[Files &amp;amp; Storage| Files &amp;amp; Storage]]&amp;lt;br &amp;gt;&lt;br /&gt;
:* [[Backups| Backups]]&amp;lt;br &amp;gt;&lt;br /&gt;
:* [[Compiling Your Code| Compiling Your Code]]&amp;lt;br &amp;gt;&lt;br /&gt;
:* [[Running Your Code| Running Your Code]]&amp;lt;br &amp;gt;&lt;br /&gt;
:* [[Graphical Applications| Graphical Applications]]&amp;lt;br &amp;gt;&lt;br /&gt;
&lt;br /&gt;
TRACC Cluster Software&lt;br /&gt;
:* [[TRACC Cluster Software#Supported On ARROW| Supported On ARROW]]&amp;lt;br &amp;gt;&lt;br /&gt;
:* [[TRACC Cluster Software#Under Investigation| Under Investigation]]&amp;lt;br &amp;gt;&lt;br /&gt;
&lt;br /&gt;
TRACC Policies&lt;br /&gt;
:* [[TRACC Policies#Application Restrictions| Application Restrictions]]&amp;lt;br &amp;gt;&lt;br /&gt;
:* [[TRACC Policies#Data Retention| Data Retention]]&amp;lt;br &amp;gt;&lt;br /&gt;
:* [[TRACC Policies#Email| Email]]&amp;lt;br &amp;gt;&lt;br /&gt;
:* [[TRACC Policies#Login Nodes| Login Nodes]]&amp;lt;br &amp;gt;&lt;/div&gt;</summary>
		<author><name>Ley</name></author>
	</entry>
	<entry>
		<id>https://wiki.anl.gov/wiki_tracc/index.php?title=Main_Page&amp;diff=2481</id>
		<title>Main Page</title>
		<link rel="alternate" type="text/html" href="https://wiki.anl.gov/wiki_tracc/index.php?title=Main_Page&amp;diff=2481"/>
		<updated>2025-12-09T06:21:15Z</updated>

		<summary type="html">&lt;p&gt;Ley: /* StarCCM+ */&lt;/p&gt;
&lt;hr /&gt;
&lt;div&gt;&amp;lt;!--&lt;br /&gt;
&lt;br /&gt;
__NOTOC__&lt;br /&gt;
{| style=&amp;quot;width:100%; background:#7CA0C4; color:white&amp;quot; border=&amp;quot;0&amp;quot; cellpadding=&amp;quot;0&amp;quot; cellspacing=&amp;quot;0&amp;quot;&lt;br /&gt;
| align=&amp;quot;left&amp;quot; width=&amp;quot;99%&amp;quot;|&lt;br /&gt;
&amp;lt;font color=&amp;quot;white&amp;quot; size=&amp;quot;+1&amp;quot;&amp;gt;&amp;lt;blockquote&amp;gt;The Transportation Research and Analysis Computing Center (TRACC)&amp;lt;/blockquote&amp;gt;&amp;lt;/font&amp;gt;&amp;lt;p&amp;gt;&lt;br /&gt;
&lt;br /&gt;
&amp;lt;font color=&amp;quot;white&amp;quot;&amp;gt;&amp;lt;blockquote&amp;gt;&lt;br /&gt;
This TRACC wiki is an EXTERNAL Argonne collaboration site to support information exchange and collaboration among TRACC users, collaborators, and partners.  The purpose of this Wiki is to: &lt;br /&gt;
* Document the setup, operation, and FAQs about TRACC supported software and hardware resources. &lt;br /&gt;
* Maintain a repository for hardware and software improvements and modifications. &lt;br /&gt;
* Share TRACC technical staff, collaborator and user generated &amp;quot;How-To&amp;quot; procedures with the transportation research and development community.&lt;br /&gt;
| style=&amp;quot;width:1%&amp;quot; | [[Image:Best.JPG]]&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
--&amp;gt;&lt;br /&gt;
=The ARROW HPC Cluster at TRACC=&lt;br /&gt;
&amp;lt;I&amp;gt;Operated by Argonne National Laboratory for the U.S. Department of Transportation&amp;lt;/I&amp;gt;&lt;br /&gt;
==Introduction==&lt;br /&gt;
The ARROW HPC cluster at TRACC is being operated for the Federal Highway Administration (FHWA) and the National Highway Traffic Safety Administration (NTHSA) at the US Department of Transportation. The site operates at Argonne National Laboratory under the Transportation Research and Analysis Computing Center (TRACC). The site is funded by specific US Department of Transportation projects and is not available to the general research community.&lt;br /&gt;
&lt;br /&gt;
TRACC has combined the original hardware from the Phoenix and Zephyr clusters into a single HPC cluster called ARROW. This consolidation allows for the efficient administration of TRACC cluster services. To mitigate the problems with load balancing, the different types of hardware nodes on the ARROW cluster are partitioned into queues with specific characteristics. When new hardware is installed to expand cluster resources, it will be made typically available via a new queue.&lt;br /&gt;
&lt;br /&gt;
ARROW is arranged such that there is a set of four login nodes that are accessed by users using the SSH protocol. The login servers are configured identically so that it doesn&#039;t matter which login server a user is connected to (the caveat is that the user should connect to the same login server where he may have created a GUI desktop session). All servers and nodes share a single parallel file system, and an individual home directory is provided for each user to store their data.&lt;br /&gt;
&lt;br /&gt;
Some common software packages provided on the cluster are commercial engineering applications such as StarCCM+, and LS-Dyna. In addition, selected open source software is available, such as OpenFOAM. Specific versions of each software are selected by the user through the &amp;quot;module&amp;quot; command, which is commonly used on HPC clusters. Connections are limited to the SSH protocol at this time, and cyber security is a major aspect of our operation. We will work with you to optimize your remote operating experience. Users will be able to create and use GUI desktops using the XFCE software. These desktops will stay active until the users decides to log out. Otherwise, GUI sessions will stay active and can be reconnected to, similar to the Windows Remote Desktop software. Thus, users can connect and disconnect from these sessions at will, and network interruptions will typically not interfere with your work flow. The PBS job submission framework allows users to use HPC resources in an efficient manner, sharing a fairly complex selection of specialty systems. This includes typical HPC nodes, as well as selected GPU systems for AI/ML applications.&lt;br /&gt;
&lt;br /&gt;
The ARROW HPC cluster consists of about 200 servers and provides a substantial computing facility and research platform for USDOT, especially in the areas of Bridge and River Hydraulics as well as Vehicle Occupant Safety analysis. It has been optimized for HPC engineering applications in Computational Fluid Dynamics and Computational Structural Mechanics. &lt;br /&gt;
&amp;lt;!--&lt;br /&gt;
:* [[introduction | &#039;&#039;&#039;Introduction to the ARROW Cluster&#039;&#039;&#039;]]&amp;lt;br &amp;gt;&lt;br /&gt;
:* [[ARROW Cluster#ARROW Queues | &#039;&#039;&#039;ARROW Computing Queues&#039;&#039;&#039;]]&amp;lt;br &amp;gt;&lt;br /&gt;
:* [[Becoming A User| &#039;&#039;&#039;Becoming A User&#039;&#039;&#039;]]&amp;lt;br &amp;gt;&lt;br /&gt;
--&amp;gt;&lt;br /&gt;
&lt;br /&gt;
==User Accounts==&lt;br /&gt;
User accounts are granted to users identified by the U.S. Department of Transportation. The current sponsors for our computing facility are the Federal Highway Administration (FHWA) and the National Highway Traffic Safety Administration (NTHSA) at the US Department of Transportation. You may want to contact [mailto:TRACC_Director@anl.gov Dr. Hubert Ley] at Argonne National Laboratory to inquire about the potential support and use of the facility for one of your transportation research projects.&lt;br /&gt;
&lt;br /&gt;
If you are working on a project that is already sponsoring the facility, have your contact person at USDOT request an account for you. We will then send you a link to a form where you can enter all the relevant contact information, and we will then create the account for you. These accounts usually start with &amp;quot;ac.&amp;quot;, for example &amp;quot;ac.username&amp;quot;, and you will need to remember that when you log in to ARROW. We will communicate all details with you once the application progresses.&lt;br /&gt;
&lt;br /&gt;
==Using the Cluster==&lt;br /&gt;
&lt;br /&gt;
The following links can be used to find more detailed information on the various aspects of the ARROW cluster&#039;s design and operation. This includes information on how to access the cluster remotely, typical applications that can be used to access the cluster, the overall layout of working queues and the job submission procedure, aspects of running specific software applications, information about software compilations, selecting software modules, using graphical applications and user interfaces, information about redundancy and backups, and much more.&lt;br /&gt;
&lt;br /&gt;
:* [[How to Connect| How to Connect]]&amp;lt;br&amp;gt;&lt;br /&gt;
:* [[Transferring Files| Transferring Files]]&amp;lt;br&amp;gt;&lt;br /&gt;
:* [[Job Submission and Monitoring| Job Submission and Monitoring]]&amp;lt;br &amp;gt;&lt;br /&gt;
&lt;br /&gt;
==Software Application Documentation==&lt;br /&gt;
&lt;br /&gt;
The following links will not work when reading these page with an external browser. They are only available when viewing the pages within the Linux desktop GUI after being logged in.&lt;br /&gt;
&lt;br /&gt;
===StarCCM+===&lt;br /&gt;
&lt;br /&gt;
* Version 20.06.007-R8&lt;br /&gt;
** [http://146.139.58.10/starccm/20.06.007-R8/STAR-CCM+20.06.007-R8/doc/en/online/STARCCMP StarCCM+ Online User&#039;s Guide (English)]&lt;br /&gt;
** [http://146.139.58.10/starccm/20.06.007-R8/STAR-CCM+20.06.007-R8/doc/zh/online/STARCCMP StarCCM+ Online User&#039;s Guide (Chinese)]&lt;br /&gt;
** [http://146.139.58.10/starccm/20.06.007-R8/STAR-CCM+20.06.007-R8/doc/api/html StarCCM+ API Documentation]&lt;br /&gt;
** [http://146.139.58.10/starccm/20.06.007-R8/STAR-CCM+20.06.007-R8/doc/client/html StarCCM+ Java API Documentation]&lt;br /&gt;
** [http://146.139.58.10/starccm/20.06.007-R8/STAR-CCM+20.06.007-R8/doc/ StarCCM+ All Manuals and Release Notes]&lt;br /&gt;
&lt;br /&gt;
* Version 20.04.007-R8&lt;br /&gt;
** [http://146.139.58.10/starccm/20.04.007-R8/STAR-CCM+20.04.007-R8/doc/en/online/STARCCMP StarCCM+ Online User&#039;s Guide (English)]&lt;br /&gt;
** [http://146.139.58.10/starccm/20.04.007-R8/STAR-CCM+20.04.007-R8/doc/zh/online/STARCCMP StarCCM+ Online User&#039;s Guide (Chinese)]&lt;br /&gt;
** [http://146.139.58.10/starccm/20.04.007-R8/STAR-CCM+20.04.007-R8/doc/api/html StarCCM+ API Documentation]&lt;br /&gt;
** [http://146.139.58.10/starccm/20.04.007-R8/STAR-CCM+20.04.007-R8/doc/client/html StarCCM+ Java API Documentation]&lt;br /&gt;
** [http://146.139.58.10/starccm/20.04.007-R8/STAR-CCM+20.04.007-R8/doc/ StarCCM+ All Manuals and Release Notes]&lt;br /&gt;
&lt;br /&gt;
* Version 19.02.009-R8&lt;br /&gt;
** [http://146.139.58.10/starccm/19.02.009-R8/STAR-CCM+19.02.009-R8/doc/en/online/STARCCMP StarCCM+ Online User&#039;s Guide (English)]&lt;br /&gt;
** [http://146.139.58.10/starccm/19.02.009-R8/STAR-CCM+19.02.009-R8/doc/zh/online/STARCCMP StarCCM+ Online User&#039;s Guide (Chinese)]&lt;br /&gt;
** [http://146.139.58.10/starccm/19.02.009-R8/STAR-CCM+19.02.009-R8/doc/api/html StarCCM+ API Documentation]&lt;br /&gt;
** [http://146.139.58.10/starccm/19.02.009-R8/STAR-CCM+19.02.009-R8/doc/client/html StarCCM+ Java API Documentation]&lt;br /&gt;
** [http://146.139.58.10/starccm/19.02.009-R8/STAR-CCM+19.02.009-R8/doc/ StarCCM+ All Manuals and Release Notes]&lt;br /&gt;
&lt;br /&gt;
* Version 18.06.006-R8&lt;br /&gt;
** [http://146.139.58.10/starccm/18.06.006-R8/STAR-CCM+18.06.006-R8/doc/en/online/STARCCMP StarCCM+ Online User&#039;s Guide (English)]&lt;br /&gt;
** [http://146.139.58.10/starccm/18.06.006-R8/STAR-CCM+18.06.006-R8/doc/zh/online/STARCCMP StarCCM+ Online User&#039;s Guide (Chinese)]&lt;br /&gt;
** [http://146.139.58.10/starccm/18.06.006-R8/STAR-CCM+18.06.006-R8/doc/api/html StarCCM+ API Documentation]&lt;br /&gt;
** [http://146.139.58.10/starccm/18.06.006-R8/STAR-CCM+18.06.006-R8/doc/client/html StarCCM+ Java API Documentation]&lt;br /&gt;
** [http://146.139.58.10/starccm/18.06.006-R8/STAR-CCM+18.06.006-R8/doc/ StarCCM+ All Manuals and Release Notes]&lt;br /&gt;
&lt;br /&gt;
* Version 18.04.008-R8&lt;br /&gt;
** [http://146.139.58.10/starccm/18.04.008-R8/STAR-CCM+18.04.008-R8/doc/en/online/STARCCMP StarCCM+ Online User&#039;s Guide (English)]&lt;br /&gt;
** [http://146.139.58.10/starccm/18.04.008-R8/STAR-CCM+18.04.008-R8/doc/zh/online/STARCCMP StarCCM+ Online User&#039;s Guide (Chinese)]&lt;br /&gt;
** [http://146.139.58.10/starccm/18.04.008-R8/STAR-CCM+18.04.008-R8/doc/api/html StarCCM+ API Documentation]&lt;br /&gt;
** [http://146.139.58.10/starccm/18.04.008-R8/STAR-CCM+18.04.008-R8/doc/client/html StarCCM+ Java API Documentation]&lt;br /&gt;
** [http://146.139.58.10/starccm/18.04.008-R8/STAR-CCM+18.04.008-R8/doc/ StarCCM+ All Manuals and Release Notes]&lt;br /&gt;
&lt;br /&gt;
* Version 17.06.007-R8&lt;br /&gt;
** [http://146.139.58.10/starccm/17.06.007-R8/STAR-CCM+17.06.007-R8/doc/en/online/STARCCMP StarCCM+ Online User&#039;s Guide (English)]&lt;br /&gt;
** [http://146.139.58.10/starccm/17.06.007-R8/STAR-CCM+17.06.007-R8/doc/zh/online/STARCCMP StarCCM+ Online User&#039;s Guide (Chinese)]&lt;br /&gt;
** [http://146.139.58.10/starccm/17.06.007-R8/STAR-CCM+17.06.007-R8/doc/api/html StarCCM+ API Documentation]&lt;br /&gt;
** [http://146.139.58.10/starccm/17.06.007-R8/STAR-CCM+17.06.007-R8/doc/client/html StarCCM+ Java API Documentation]&lt;br /&gt;
** [http://146.139.58.10/starccm/17.06.007-R8/STAR-CCM+17.06.007-R8/doc/ StarCCM+ All Manuals and Release Notes]&lt;br /&gt;
&lt;br /&gt;
* Version 17.04.007-R8&lt;br /&gt;
** [http://146.139.58.10/starccm/17.04.007-R8/STAR-CCM+17.04.007-R8/doc/en/online/STARCCMP StarCCM+ Online User&#039;s Guide (English)]&lt;br /&gt;
** [http://146.139.58.10/starccm/17.04.007-R8/STAR-CCM+17.04.007-R8/doc/zh/online/STARCCMP StarCCM+ Online User&#039;s Guide (Chinese)]&lt;br /&gt;
** [http://146.139.58.10/starccm/17.04.007-R8/STAR-CCM+17.04.007-R8/doc/api/html StarCCM+ API Documentation]&lt;br /&gt;
** [http://146.139.58.10/starccm/17.04.007-R8/STAR-CCM+17.04.007-R8/doc/client/html StarCCM+ Java API Documentation]&lt;br /&gt;
** [http://146.139.58.10/starccm/17.04.007-R8/STAR-CCM+17.04.007-R8/doc/ StarCCM+ All Manuals and Release Notes]&lt;br /&gt;
&lt;br /&gt;
* Version 17.02.008-R8&lt;br /&gt;
** [http://146.139.58.10/starccm/17.02.008-R8/STAR-CCM+17.02.008-R8/doc/en/online/STARCCMP StarCCM+ Online User&#039;s Guide (English)]&lt;br /&gt;
** [http://146.139.58.10/starccm/17.02.008-R8/STAR-CCM+17.02.008-R8/doc/zh/online/STARCCMP StarCCM+ Online User&#039;s Guide (Chinese)]&lt;br /&gt;
** [http://146.139.58.10/starccm/17.02.008-R8/STAR-CCM+17.02.008-R8/doc/api/html StarCCM+ API Documentation]&lt;br /&gt;
** [http://146.139.58.10/starccm/17.02.008-R8/STAR-CCM+17.02.008-R8/doc/client/html StarCCM+ Java API Documentation]&lt;br /&gt;
** [http://146.139.58.10/starccm/17.02.008-R8/STAR-CCM+17.02.008-R8/doc/ StarCCM+ All Manuals and Release Notes]&lt;br /&gt;
&lt;br /&gt;
* Version 17.02.007-R8&lt;br /&gt;
** [http://146.139.58.10/starccm/17.02.007-R8/STAR-CCM+17.02.007-R8/doc/en/online/STARCCMP StarCCM+ Online User&#039;s Guide (English)]&lt;br /&gt;
** [http://146.139.58.10/starccm/17.02.007-R8/STAR-CCM+17.02.007-R8/doc/zh/online/STARCCMP StarCCM+ Online User&#039;s Guide (Chinese)]&lt;br /&gt;
** [http://146.139.58.10/starccm/17.02.007-R8/STAR-CCM+17.02.007-R8/doc/api/html StarCCM+ API Documentation]&lt;br /&gt;
** [http://146.139.58.10/starccm/17.02.007-R8/STAR-CCM+17.02.007-R8/doc/client/html StarCCM+ Java API Documentation]&lt;br /&gt;
** [http://146.139.58.10/starccm/17.02.007-R8/STAR-CCM+17.02.007-R8/doc/ StarCCM+ All Manuals and Release Notes]&lt;br /&gt;
&lt;br /&gt;
===LS-Dyna+===&lt;br /&gt;
&lt;br /&gt;
* LS-Dyna Version R16&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R16/LS-DYNA_Manual_Theory_R16.pdf LS-Dyna Theory Manual]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R16/LS-DYNA_Manual_Vol_I_R16.pdf LS-Dyna User&#039;s Manual Vol 1]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R16/LS-DYNA_Manual_Vol_II_R16.pdf LS-Dyna User&#039;s Manual Vol 2]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R16/LS-DYNA_Manual_Vol_III_R16.pdf LS-Dyna User&#039;s Manual Vol 3]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R16/ All Manuals and Release Notes]&lt;br /&gt;
&lt;br /&gt;
* LS-Dyna Version R15&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R15/LS-DYNA_Manual_Theory_R15.pdf LS-Dyna Theory Manual]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R15/LS-DYNA_Manual_Volume_I_R15.pdf LS-Dyna User&#039;s Manual Vol 1]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R15/LS-DYNA_Manual_Volume_II_R15.pdf LS-Dyna User&#039;s Manual Vol 2]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R15/LS-DYNA_Manual_Volume_III_R15.pdf LS-Dyna User&#039;s Manual Vol 3]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R15/ All Manuals and Release Notes]&lt;br /&gt;
&lt;br /&gt;
* LS-Dyna Version R14&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R14/LS-DYNA_Manual_Theory_R14.pdf LS-Dyna Theory Manual]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R14/LS-DYNA_Manual_Volume_I_R14.pdf LS-Dyna User&#039;s Manual Vol 1]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R14/LS-DYNA_Manual_Volume_II_R14.pdf LS-Dyna User&#039;s Manual Vol 2]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R14/LS-DYNA_Manual_Volume_III_R14.pdf LS-Dyna User&#039;s Manual Vol 3]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R14/ All Manuals and Release Notes]&lt;br /&gt;
&lt;br /&gt;
* LS-Dyna Version R13&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R13/LS-DYNA_Manual_Volume_I_R13.pdf LS-Dyna User&#039;s Manual Vol 1]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R13/LS-DYNA_Manual_Volume_II_R13.pdf LS-Dyna User&#039;s Manual Vol 2]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R13/LS-DYNA_Manual_Volume_III_R13.pdf LS-Dyna User&#039;s Manual Vol 3]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R13/ All Manuals and Release Notes]&lt;br /&gt;
&lt;br /&gt;
* LS-Dyna Version R12&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R12/LS-DYNA_Manual_Volume_I_R12.pdf LS-Dyna User&#039;s Manual Vol 1]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R12/LS-DYNA_Manual_Volume_II_R12.pdf LS-Dyna User&#039;s Manual Vol 2]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R12/LS-DYNA_Manual_Volume_III_R12.pdf LS-Dyna User&#039;s Manual Vol 3]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R12/ All Manuals and Release Notes]&lt;br /&gt;
&lt;br /&gt;
* LS-Dyna Version R11&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R11/LS-DYNA_Manual_Volume_I_R11.pdf LS-Dyna User&#039;s Manual Vol 1]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R11/LS-DYNA_Manual_Volume_II_R11.pdf LS-Dyna User&#039;s Manual Vol 2]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R11/LS-DYNA_Manual_Volume_III_R11.pdf LS-Dyna User&#039;s Manual Vol 3]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R11/ All Manuals and Release Notes]&lt;br /&gt;
&lt;br /&gt;
=The links and pages shown below are outdated!=&lt;br /&gt;
&lt;br /&gt;
&amp;lt;BLOCKQUOTE&amp;gt;&lt;br /&gt;
[[File:Attention.jpg|25px]] &#039;&#039;&#039;IMPORTANT NOTE:&#039;&#039;&#039; &#039;&#039;We are updating the wiki pages with the most recent information on services, hardware, and capabilities. The information provided below is outdated and in urgent need for a update. We recommend to disregard the information for now.&#039;&#039;&lt;br /&gt;
&amp;lt;/BLOCKQUOTE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
General Information&lt;br /&gt;
:* [[Becoming_A_User| Becoming_A_User]]&amp;lt;br &amp;gt;&lt;br /&gt;
&lt;br /&gt;
Procedures&lt;br /&gt;
:* [[Transferring Files| Transferring Files]]&amp;lt;br &amp;gt;&lt;br /&gt;
:* [[Setting Up Your Environment| Setting Up Your Environment]]&amp;lt;br &amp;gt;&lt;br /&gt;
:* [[Login Nodes Versus Compute Nodes| Login Nodes Versus Compute Nodes]]&amp;lt;br &amp;gt;&lt;br /&gt;
:* [[Files &amp;amp; Storage| Files &amp;amp; Storage]]&amp;lt;br &amp;gt;&lt;br /&gt;
:* [[Backups| Backups]]&amp;lt;br &amp;gt;&lt;br /&gt;
:* [[Compiling Your Code| Compiling Your Code]]&amp;lt;br &amp;gt;&lt;br /&gt;
:* [[Running Your Code| Running Your Code]]&amp;lt;br &amp;gt;&lt;br /&gt;
:* [[Graphical Applications| Graphical Applications]]&amp;lt;br &amp;gt;&lt;br /&gt;
&lt;br /&gt;
TRACC Cluster Software&lt;br /&gt;
:* [[TRACC Cluster Software#Supported On ARROW| Supported On ARROW]]&amp;lt;br &amp;gt;&lt;br /&gt;
:* [[TRACC Cluster Software#Under Investigation| Under Investigation]]&amp;lt;br &amp;gt;&lt;br /&gt;
&lt;br /&gt;
TRACC Policies&lt;br /&gt;
:* [[TRACC Policies#Application Restrictions| Application Restrictions]]&amp;lt;br &amp;gt;&lt;br /&gt;
:* [[TRACC Policies#Data Retention| Data Retention]]&amp;lt;br &amp;gt;&lt;br /&gt;
:* [[TRACC Policies#Email| Email]]&amp;lt;br &amp;gt;&lt;br /&gt;
:* [[TRACC Policies#Login Nodes| Login Nodes]]&amp;lt;br &amp;gt;&lt;/div&gt;</summary>
		<author><name>Ley</name></author>
	</entry>
	<entry>
		<id>https://wiki.anl.gov/wiki_tracc/index.php?title=Main_Page&amp;diff=2480</id>
		<title>Main Page</title>
		<link rel="alternate" type="text/html" href="https://wiki.anl.gov/wiki_tracc/index.php?title=Main_Page&amp;diff=2480"/>
		<updated>2025-12-09T06:04:35Z</updated>

		<summary type="html">&lt;p&gt;Ley: /* StarCCM+ */&lt;/p&gt;
&lt;hr /&gt;
&lt;div&gt;&amp;lt;!--&lt;br /&gt;
&lt;br /&gt;
__NOTOC__&lt;br /&gt;
{| style=&amp;quot;width:100%; background:#7CA0C4; color:white&amp;quot; border=&amp;quot;0&amp;quot; cellpadding=&amp;quot;0&amp;quot; cellspacing=&amp;quot;0&amp;quot;&lt;br /&gt;
| align=&amp;quot;left&amp;quot; width=&amp;quot;99%&amp;quot;|&lt;br /&gt;
&amp;lt;font color=&amp;quot;white&amp;quot; size=&amp;quot;+1&amp;quot;&amp;gt;&amp;lt;blockquote&amp;gt;The Transportation Research and Analysis Computing Center (TRACC)&amp;lt;/blockquote&amp;gt;&amp;lt;/font&amp;gt;&amp;lt;p&amp;gt;&lt;br /&gt;
&lt;br /&gt;
&amp;lt;font color=&amp;quot;white&amp;quot;&amp;gt;&amp;lt;blockquote&amp;gt;&lt;br /&gt;
This TRACC wiki is an EXTERNAL Argonne collaboration site to support information exchange and collaboration among TRACC users, collaborators, and partners.  The purpose of this Wiki is to: &lt;br /&gt;
* Document the setup, operation, and FAQs about TRACC supported software and hardware resources. &lt;br /&gt;
* Maintain a repository for hardware and software improvements and modifications. &lt;br /&gt;
* Share TRACC technical staff, collaborator and user generated &amp;quot;How-To&amp;quot; procedures with the transportation research and development community.&lt;br /&gt;
| style=&amp;quot;width:1%&amp;quot; | [[Image:Best.JPG]]&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
--&amp;gt;&lt;br /&gt;
=The ARROW HPC Cluster at TRACC=&lt;br /&gt;
&amp;lt;I&amp;gt;Operated by Argonne National Laboratory for the U.S. Department of Transportation&amp;lt;/I&amp;gt;&lt;br /&gt;
==Introduction==&lt;br /&gt;
The ARROW HPC cluster at TRACC is being operated for the Federal Highway Administration (FHWA) and the National Highway Traffic Safety Administration (NTHSA) at the US Department of Transportation. The site operates at Argonne National Laboratory under the Transportation Research and Analysis Computing Center (TRACC). The site is funded by specific US Department of Transportation projects and is not available to the general research community.&lt;br /&gt;
&lt;br /&gt;
TRACC has combined the original hardware from the Phoenix and Zephyr clusters into a single HPC cluster called ARROW. This consolidation allows for the efficient administration of TRACC cluster services. To mitigate the problems with load balancing, the different types of hardware nodes on the ARROW cluster are partitioned into queues with specific characteristics. When new hardware is installed to expand cluster resources, it will be made typically available via a new queue.&lt;br /&gt;
&lt;br /&gt;
ARROW is arranged such that there is a set of four login nodes that are accessed by users using the SSH protocol. The login servers are configured identically so that it doesn&#039;t matter which login server a user is connected to (the caveat is that the user should connect to the same login server where he may have created a GUI desktop session). All servers and nodes share a single parallel file system, and an individual home directory is provided for each user to store their data.&lt;br /&gt;
&lt;br /&gt;
Some common software packages provided on the cluster are commercial engineering applications such as StarCCM+, and LS-Dyna. In addition, selected open source software is available, such as OpenFOAM. Specific versions of each software are selected by the user through the &amp;quot;module&amp;quot; command, which is commonly used on HPC clusters. Connections are limited to the SSH protocol at this time, and cyber security is a major aspect of our operation. We will work with you to optimize your remote operating experience. Users will be able to create and use GUI desktops using the XFCE software. These desktops will stay active until the users decides to log out. Otherwise, GUI sessions will stay active and can be reconnected to, similar to the Windows Remote Desktop software. Thus, users can connect and disconnect from these sessions at will, and network interruptions will typically not interfere with your work flow. The PBS job submission framework allows users to use HPC resources in an efficient manner, sharing a fairly complex selection of specialty systems. This includes typical HPC nodes, as well as selected GPU systems for AI/ML applications.&lt;br /&gt;
&lt;br /&gt;
The ARROW HPC cluster consists of about 200 servers and provides a substantial computing facility and research platform for USDOT, especially in the areas of Bridge and River Hydraulics as well as Vehicle Occupant Safety analysis. It has been optimized for HPC engineering applications in Computational Fluid Dynamics and Computational Structural Mechanics. &lt;br /&gt;
&amp;lt;!--&lt;br /&gt;
:* [[introduction | &#039;&#039;&#039;Introduction to the ARROW Cluster&#039;&#039;&#039;]]&amp;lt;br &amp;gt;&lt;br /&gt;
:* [[ARROW Cluster#ARROW Queues | &#039;&#039;&#039;ARROW Computing Queues&#039;&#039;&#039;]]&amp;lt;br &amp;gt;&lt;br /&gt;
:* [[Becoming A User| &#039;&#039;&#039;Becoming A User&#039;&#039;&#039;]]&amp;lt;br &amp;gt;&lt;br /&gt;
--&amp;gt;&lt;br /&gt;
&lt;br /&gt;
==User Accounts==&lt;br /&gt;
User accounts are granted to users identified by the U.S. Department of Transportation. The current sponsors for our computing facility are the Federal Highway Administration (FHWA) and the National Highway Traffic Safety Administration (NTHSA) at the US Department of Transportation. You may want to contact [mailto:TRACC_Director@anl.gov Dr. Hubert Ley] at Argonne National Laboratory to inquire about the potential support and use of the facility for one of your transportation research projects.&lt;br /&gt;
&lt;br /&gt;
If you are working on a project that is already sponsoring the facility, have your contact person at USDOT request an account for you. We will then send you a link to a form where you can enter all the relevant contact information, and we will then create the account for you. These accounts usually start with &amp;quot;ac.&amp;quot;, for example &amp;quot;ac.username&amp;quot;, and you will need to remember that when you log in to ARROW. We will communicate all details with you once the application progresses.&lt;br /&gt;
&lt;br /&gt;
==Using the Cluster==&lt;br /&gt;
&lt;br /&gt;
The following links can be used to find more detailed information on the various aspects of the ARROW cluster&#039;s design and operation. This includes information on how to access the cluster remotely, typical applications that can be used to access the cluster, the overall layout of working queues and the job submission procedure, aspects of running specific software applications, information about software compilations, selecting software modules, using graphical applications and user interfaces, information about redundancy and backups, and much more.&lt;br /&gt;
&lt;br /&gt;
:* [[How to Connect| How to Connect]]&amp;lt;br&amp;gt;&lt;br /&gt;
:* [[Transferring Files| Transferring Files]]&amp;lt;br&amp;gt;&lt;br /&gt;
:* [[Job Submission and Monitoring| Job Submission and Monitoring]]&amp;lt;br &amp;gt;&lt;br /&gt;
&lt;br /&gt;
==Software Application Documentation==&lt;br /&gt;
&lt;br /&gt;
The following links will not work when reading these page with an external browser. They are only available when viewing the pages within the Linux desktop GUI after being logged in.&lt;br /&gt;
&lt;br /&gt;
===StarCCM+===&lt;br /&gt;
&lt;br /&gt;
* Version 20.04.007-R8&lt;br /&gt;
** [http://146.139.58.10/starccm/20.06.007-R8/STAR-CCM+20.06.007-R8/doc/en/online/STARCCMP StarCCM+ Online User&#039;s Guide (English)]&lt;br /&gt;
** [http://146.139.58.10/starccm/20.06.007-R8/STAR-CCM+20.06.007-R8/doc/zh/online/STARCCMP StarCCM+ Online User&#039;s Guide (Chinese)]&lt;br /&gt;
** [http://146.139.58.10/starccm/20.06.007-R8/STAR-CCM+20.06.007-R8/doc/api/html StarCCM+ API Documentation]&lt;br /&gt;
** [http://146.139.58.10/starccm/20.06.007-R8/STAR-CCM+20.06.007-R8/doc/client/html StarCCM+ Java API Documentation]&lt;br /&gt;
** [http://146.139.58.10/starccm/20.06.007-R8/STAR-CCM+20.06.007-R8/doc/ StarCCM+ All Manuals and Release Notes]&lt;br /&gt;
&lt;br /&gt;
* Version 20.04.007-R8&lt;br /&gt;
** [http://146.139.58.10/starccm/20.04.007-R8/STAR-CCM+20.04.007-R8/doc/en/online/STARCCMP StarCCM+ Online User&#039;s Guide (English)]&lt;br /&gt;
** [http://146.139.58.10/starccm/20.04.007-R8/STAR-CCM+20.04.007-R8/doc/zh/online/STARCCMP StarCCM+ Online User&#039;s Guide (Chinese)]&lt;br /&gt;
** [http://146.139.58.10/starccm/20.04.007-R8/STAR-CCM+20.04.007-R8/doc/api/html StarCCM+ API Documentation]&lt;br /&gt;
** [http://146.139.58.10/starccm/20.04.007-R8/STAR-CCM+20.04.007-R8/doc/client/html StarCCM+ Java API Documentation]&lt;br /&gt;
** [http://146.139.58.10/starccm/20.04.007-R8/STAR-CCM+20.04.007-R8/doc/ StarCCM+ All Manuals and Release Notes]&lt;br /&gt;
&lt;br /&gt;
===LS-Dyna+===&lt;br /&gt;
&lt;br /&gt;
* LS-Dyna Version R16&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R16/LS-DYNA_Manual_Theory_R16.pdf LS-Dyna Theory Manual]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R16/LS-DYNA_Manual_Vol_I_R16.pdf LS-Dyna User&#039;s Manual Vol 1]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R16/LS-DYNA_Manual_Vol_II_R16.pdf LS-Dyna User&#039;s Manual Vol 2]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R16/LS-DYNA_Manual_Vol_III_R16.pdf LS-Dyna User&#039;s Manual Vol 3]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R16/ All Manuals and Release Notes]&lt;br /&gt;
&lt;br /&gt;
* LS-Dyna Version R15&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R15/LS-DYNA_Manual_Theory_R15.pdf LS-Dyna Theory Manual]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R15/LS-DYNA_Manual_Volume_I_R15.pdf LS-Dyna User&#039;s Manual Vol 1]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R15/LS-DYNA_Manual_Volume_II_R15.pdf LS-Dyna User&#039;s Manual Vol 2]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R15/LS-DYNA_Manual_Volume_III_R15.pdf LS-Dyna User&#039;s Manual Vol 3]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R15/ All Manuals and Release Notes]&lt;br /&gt;
&lt;br /&gt;
* LS-Dyna Version R14&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R14/LS-DYNA_Manual_Theory_R14.pdf LS-Dyna Theory Manual]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R14/LS-DYNA_Manual_Volume_I_R14.pdf LS-Dyna User&#039;s Manual Vol 1]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R14/LS-DYNA_Manual_Volume_II_R14.pdf LS-Dyna User&#039;s Manual Vol 2]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R14/LS-DYNA_Manual_Volume_III_R14.pdf LS-Dyna User&#039;s Manual Vol 3]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R14/ All Manuals and Release Notes]&lt;br /&gt;
&lt;br /&gt;
* LS-Dyna Version R13&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R13/LS-DYNA_Manual_Volume_I_R13.pdf LS-Dyna User&#039;s Manual Vol 1]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R13/LS-DYNA_Manual_Volume_II_R13.pdf LS-Dyna User&#039;s Manual Vol 2]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R13/LS-DYNA_Manual_Volume_III_R13.pdf LS-Dyna User&#039;s Manual Vol 3]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R13/ All Manuals and Release Notes]&lt;br /&gt;
&lt;br /&gt;
* LS-Dyna Version R12&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R12/LS-DYNA_Manual_Volume_I_R12.pdf LS-Dyna User&#039;s Manual Vol 1]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R12/LS-DYNA_Manual_Volume_II_R12.pdf LS-Dyna User&#039;s Manual Vol 2]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R12/LS-DYNA_Manual_Volume_III_R12.pdf LS-Dyna User&#039;s Manual Vol 3]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R12/ All Manuals and Release Notes]&lt;br /&gt;
&lt;br /&gt;
* LS-Dyna Version R11&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R11/LS-DYNA_Manual_Volume_I_R11.pdf LS-Dyna User&#039;s Manual Vol 1]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R11/LS-DYNA_Manual_Volume_II_R11.pdf LS-Dyna User&#039;s Manual Vol 2]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R11/LS-DYNA_Manual_Volume_III_R11.pdf LS-Dyna User&#039;s Manual Vol 3]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R11/ All Manuals and Release Notes]&lt;br /&gt;
&lt;br /&gt;
=The links and pages shown below are outdated!=&lt;br /&gt;
&lt;br /&gt;
&amp;lt;BLOCKQUOTE&amp;gt;&lt;br /&gt;
[[File:Attention.jpg|25px]] &#039;&#039;&#039;IMPORTANT NOTE:&#039;&#039;&#039; &#039;&#039;We are updating the wiki pages with the most recent information on services, hardware, and capabilities. The information provided below is outdated and in urgent need for a update. We recommend to disregard the information for now.&#039;&#039;&lt;br /&gt;
&amp;lt;/BLOCKQUOTE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
General Information&lt;br /&gt;
:* [[Becoming_A_User| Becoming_A_User]]&amp;lt;br &amp;gt;&lt;br /&gt;
&lt;br /&gt;
Procedures&lt;br /&gt;
:* [[Transferring Files| Transferring Files]]&amp;lt;br &amp;gt;&lt;br /&gt;
:* [[Setting Up Your Environment| Setting Up Your Environment]]&amp;lt;br &amp;gt;&lt;br /&gt;
:* [[Login Nodes Versus Compute Nodes| Login Nodes Versus Compute Nodes]]&amp;lt;br &amp;gt;&lt;br /&gt;
:* [[Files &amp;amp; Storage| Files &amp;amp; Storage]]&amp;lt;br &amp;gt;&lt;br /&gt;
:* [[Backups| Backups]]&amp;lt;br &amp;gt;&lt;br /&gt;
:* [[Compiling Your Code| Compiling Your Code]]&amp;lt;br &amp;gt;&lt;br /&gt;
:* [[Running Your Code| Running Your Code]]&amp;lt;br &amp;gt;&lt;br /&gt;
:* [[Graphical Applications| Graphical Applications]]&amp;lt;br &amp;gt;&lt;br /&gt;
&lt;br /&gt;
TRACC Cluster Software&lt;br /&gt;
:* [[TRACC Cluster Software#Supported On ARROW| Supported On ARROW]]&amp;lt;br &amp;gt;&lt;br /&gt;
:* [[TRACC Cluster Software#Under Investigation| Under Investigation]]&amp;lt;br &amp;gt;&lt;br /&gt;
&lt;br /&gt;
TRACC Policies&lt;br /&gt;
:* [[TRACC Policies#Application Restrictions| Application Restrictions]]&amp;lt;br &amp;gt;&lt;br /&gt;
:* [[TRACC Policies#Data Retention| Data Retention]]&amp;lt;br &amp;gt;&lt;br /&gt;
:* [[TRACC Policies#Email| Email]]&amp;lt;br &amp;gt;&lt;br /&gt;
:* [[TRACC Policies#Login Nodes| Login Nodes]]&amp;lt;br &amp;gt;&lt;/div&gt;</summary>
		<author><name>Ley</name></author>
	</entry>
	<entry>
		<id>https://wiki.anl.gov/wiki_tracc/index.php?title=Main_Page&amp;diff=2479</id>
		<title>Main Page</title>
		<link rel="alternate" type="text/html" href="https://wiki.anl.gov/wiki_tracc/index.php?title=Main_Page&amp;diff=2479"/>
		<updated>2025-12-09T06:00:39Z</updated>

		<summary type="html">&lt;p&gt;Ley: /* LS-Dyna+ */&lt;/p&gt;
&lt;hr /&gt;
&lt;div&gt;&amp;lt;!--&lt;br /&gt;
&lt;br /&gt;
__NOTOC__&lt;br /&gt;
{| style=&amp;quot;width:100%; background:#7CA0C4; color:white&amp;quot; border=&amp;quot;0&amp;quot; cellpadding=&amp;quot;0&amp;quot; cellspacing=&amp;quot;0&amp;quot;&lt;br /&gt;
| align=&amp;quot;left&amp;quot; width=&amp;quot;99%&amp;quot;|&lt;br /&gt;
&amp;lt;font color=&amp;quot;white&amp;quot; size=&amp;quot;+1&amp;quot;&amp;gt;&amp;lt;blockquote&amp;gt;The Transportation Research and Analysis Computing Center (TRACC)&amp;lt;/blockquote&amp;gt;&amp;lt;/font&amp;gt;&amp;lt;p&amp;gt;&lt;br /&gt;
&lt;br /&gt;
&amp;lt;font color=&amp;quot;white&amp;quot;&amp;gt;&amp;lt;blockquote&amp;gt;&lt;br /&gt;
This TRACC wiki is an EXTERNAL Argonne collaboration site to support information exchange and collaboration among TRACC users, collaborators, and partners.  The purpose of this Wiki is to: &lt;br /&gt;
* Document the setup, operation, and FAQs about TRACC supported software and hardware resources. &lt;br /&gt;
* Maintain a repository for hardware and software improvements and modifications. &lt;br /&gt;
* Share TRACC technical staff, collaborator and user generated &amp;quot;How-To&amp;quot; procedures with the transportation research and development community.&lt;br /&gt;
| style=&amp;quot;width:1%&amp;quot; | [[Image:Best.JPG]]&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
--&amp;gt;&lt;br /&gt;
=The ARROW HPC Cluster at TRACC=&lt;br /&gt;
&amp;lt;I&amp;gt;Operated by Argonne National Laboratory for the U.S. Department of Transportation&amp;lt;/I&amp;gt;&lt;br /&gt;
==Introduction==&lt;br /&gt;
The ARROW HPC cluster at TRACC is being operated for the Federal Highway Administration (FHWA) and the National Highway Traffic Safety Administration (NTHSA) at the US Department of Transportation. The site operates at Argonne National Laboratory under the Transportation Research and Analysis Computing Center (TRACC). The site is funded by specific US Department of Transportation projects and is not available to the general research community.&lt;br /&gt;
&lt;br /&gt;
TRACC has combined the original hardware from the Phoenix and Zephyr clusters into a single HPC cluster called ARROW. This consolidation allows for the efficient administration of TRACC cluster services. To mitigate the problems with load balancing, the different types of hardware nodes on the ARROW cluster are partitioned into queues with specific characteristics. When new hardware is installed to expand cluster resources, it will be made typically available via a new queue.&lt;br /&gt;
&lt;br /&gt;
ARROW is arranged such that there is a set of four login nodes that are accessed by users using the SSH protocol. The login servers are configured identically so that it doesn&#039;t matter which login server a user is connected to (the caveat is that the user should connect to the same login server where he may have created a GUI desktop session). All servers and nodes share a single parallel file system, and an individual home directory is provided for each user to store their data.&lt;br /&gt;
&lt;br /&gt;
Some common software packages provided on the cluster are commercial engineering applications such as StarCCM+, and LS-Dyna. In addition, selected open source software is available, such as OpenFOAM. Specific versions of each software are selected by the user through the &amp;quot;module&amp;quot; command, which is commonly used on HPC clusters. Connections are limited to the SSH protocol at this time, and cyber security is a major aspect of our operation. We will work with you to optimize your remote operating experience. Users will be able to create and use GUI desktops using the XFCE software. These desktops will stay active until the users decides to log out. Otherwise, GUI sessions will stay active and can be reconnected to, similar to the Windows Remote Desktop software. Thus, users can connect and disconnect from these sessions at will, and network interruptions will typically not interfere with your work flow. The PBS job submission framework allows users to use HPC resources in an efficient manner, sharing a fairly complex selection of specialty systems. This includes typical HPC nodes, as well as selected GPU systems for AI/ML applications.&lt;br /&gt;
&lt;br /&gt;
The ARROW HPC cluster consists of about 200 servers and provides a substantial computing facility and research platform for USDOT, especially in the areas of Bridge and River Hydraulics as well as Vehicle Occupant Safety analysis. It has been optimized for HPC engineering applications in Computational Fluid Dynamics and Computational Structural Mechanics. &lt;br /&gt;
&amp;lt;!--&lt;br /&gt;
:* [[introduction | &#039;&#039;&#039;Introduction to the ARROW Cluster&#039;&#039;&#039;]]&amp;lt;br &amp;gt;&lt;br /&gt;
:* [[ARROW Cluster#ARROW Queues | &#039;&#039;&#039;ARROW Computing Queues&#039;&#039;&#039;]]&amp;lt;br &amp;gt;&lt;br /&gt;
:* [[Becoming A User| &#039;&#039;&#039;Becoming A User&#039;&#039;&#039;]]&amp;lt;br &amp;gt;&lt;br /&gt;
--&amp;gt;&lt;br /&gt;
&lt;br /&gt;
==User Accounts==&lt;br /&gt;
User accounts are granted to users identified by the U.S. Department of Transportation. The current sponsors for our computing facility are the Federal Highway Administration (FHWA) and the National Highway Traffic Safety Administration (NTHSA) at the US Department of Transportation. You may want to contact [mailto:TRACC_Director@anl.gov Dr. Hubert Ley] at Argonne National Laboratory to inquire about the potential support and use of the facility for one of your transportation research projects.&lt;br /&gt;
&lt;br /&gt;
If you are working on a project that is already sponsoring the facility, have your contact person at USDOT request an account for you. We will then send you a link to a form where you can enter all the relevant contact information, and we will then create the account for you. These accounts usually start with &amp;quot;ac.&amp;quot;, for example &amp;quot;ac.username&amp;quot;, and you will need to remember that when you log in to ARROW. We will communicate all details with you once the application progresses.&lt;br /&gt;
&lt;br /&gt;
==Using the Cluster==&lt;br /&gt;
&lt;br /&gt;
The following links can be used to find more detailed information on the various aspects of the ARROW cluster&#039;s design and operation. This includes information on how to access the cluster remotely, typical applications that can be used to access the cluster, the overall layout of working queues and the job submission procedure, aspects of running specific software applications, information about software compilations, selecting software modules, using graphical applications and user interfaces, information about redundancy and backups, and much more.&lt;br /&gt;
&lt;br /&gt;
:* [[How to Connect| How to Connect]]&amp;lt;br&amp;gt;&lt;br /&gt;
:* [[Transferring Files| Transferring Files]]&amp;lt;br&amp;gt;&lt;br /&gt;
:* [[Job Submission and Monitoring| Job Submission and Monitoring]]&amp;lt;br &amp;gt;&lt;br /&gt;
&lt;br /&gt;
==Software Application Documentation==&lt;br /&gt;
&lt;br /&gt;
The following links will not work when reading these page with an external browser. They are only available when viewing the pages within the Linux desktop GUI after being logged in.&lt;br /&gt;
&lt;br /&gt;
===StarCCM+===&lt;br /&gt;
&lt;br /&gt;
* Version 20.04.007-R8&lt;br /&gt;
** [http://146.139.58.10/starccm/20.04.007-R8/STAR-CCM+20.04.007-R8/doc/en/online/STARCCMP StarCCM+ Online User&#039;s Guide (English)]&lt;br /&gt;
** [http://146.139.58.10/starccm/20.04.007-R8/STAR-CCM+20.04.007-R8/doc/zh/online/STARCCMP StarCCM+ Online User&#039;s Guide (Chinese)]&lt;br /&gt;
** [http://146.139.58.10/starccm/20.04.007-R8/STAR-CCM+20.04.007-R8/doc/api/html StarCCM+ API Documentation]&lt;br /&gt;
** [http://146.139.58.10/starccm/20.04.007-R8/STAR-CCM+20.04.007-R8/doc/client/html StarCCM+ Java API Documentation]&lt;br /&gt;
** [http://146.139.58.10/starccm/20.04.007-R8/STAR-CCM+20.04.007-R8/doc/ StarCCM+ All Manuals and Release Notes]&lt;br /&gt;
&lt;br /&gt;
===LS-Dyna+===&lt;br /&gt;
&lt;br /&gt;
* LS-Dyna Version R16&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R16/LS-DYNA_Manual_Theory_R16.pdf LS-Dyna Theory Manual]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R16/LS-DYNA_Manual_Vol_I_R16.pdf LS-Dyna User&#039;s Manual Vol 1]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R16/LS-DYNA_Manual_Vol_II_R16.pdf LS-Dyna User&#039;s Manual Vol 2]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R16/LS-DYNA_Manual_Vol_III_R16.pdf LS-Dyna User&#039;s Manual Vol 3]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R16/ All Manuals and Release Notes]&lt;br /&gt;
&lt;br /&gt;
* LS-Dyna Version R15&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R15/LS-DYNA_Manual_Theory_R15.pdf LS-Dyna Theory Manual]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R15/LS-DYNA_Manual_Volume_I_R15.pdf LS-Dyna User&#039;s Manual Vol 1]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R15/LS-DYNA_Manual_Volume_II_R15.pdf LS-Dyna User&#039;s Manual Vol 2]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R15/LS-DYNA_Manual_Volume_III_R15.pdf LS-Dyna User&#039;s Manual Vol 3]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R15/ All Manuals and Release Notes]&lt;br /&gt;
&lt;br /&gt;
* LS-Dyna Version R14&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R14/LS-DYNA_Manual_Theory_R14.pdf LS-Dyna Theory Manual]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R14/LS-DYNA_Manual_Volume_I_R14.pdf LS-Dyna User&#039;s Manual Vol 1]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R14/LS-DYNA_Manual_Volume_II_R14.pdf LS-Dyna User&#039;s Manual Vol 2]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R14/LS-DYNA_Manual_Volume_III_R14.pdf LS-Dyna User&#039;s Manual Vol 3]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R14/ All Manuals and Release Notes]&lt;br /&gt;
&lt;br /&gt;
* LS-Dyna Version R13&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R13/LS-DYNA_Manual_Volume_I_R13.pdf LS-Dyna User&#039;s Manual Vol 1]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R13/LS-DYNA_Manual_Volume_II_R13.pdf LS-Dyna User&#039;s Manual Vol 2]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R13/LS-DYNA_Manual_Volume_III_R13.pdf LS-Dyna User&#039;s Manual Vol 3]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R13/ All Manuals and Release Notes]&lt;br /&gt;
&lt;br /&gt;
* LS-Dyna Version R12&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R12/LS-DYNA_Manual_Volume_I_R12.pdf LS-Dyna User&#039;s Manual Vol 1]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R12/LS-DYNA_Manual_Volume_II_R12.pdf LS-Dyna User&#039;s Manual Vol 2]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R12/LS-DYNA_Manual_Volume_III_R12.pdf LS-Dyna User&#039;s Manual Vol 3]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R12/ All Manuals and Release Notes]&lt;br /&gt;
&lt;br /&gt;
* LS-Dyna Version R11&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R11/LS-DYNA_Manual_Volume_I_R11.pdf LS-Dyna User&#039;s Manual Vol 1]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R11/LS-DYNA_Manual_Volume_II_R11.pdf LS-Dyna User&#039;s Manual Vol 2]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R11/LS-DYNA_Manual_Volume_III_R11.pdf LS-Dyna User&#039;s Manual Vol 3]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R11/ All Manuals and Release Notes]&lt;br /&gt;
&lt;br /&gt;
=The links and pages shown below are outdated!=&lt;br /&gt;
&lt;br /&gt;
&amp;lt;BLOCKQUOTE&amp;gt;&lt;br /&gt;
[[File:Attention.jpg|25px]] &#039;&#039;&#039;IMPORTANT NOTE:&#039;&#039;&#039; &#039;&#039;We are updating the wiki pages with the most recent information on services, hardware, and capabilities. The information provided below is outdated and in urgent need for a update. We recommend to disregard the information for now.&#039;&#039;&lt;br /&gt;
&amp;lt;/BLOCKQUOTE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
General Information&lt;br /&gt;
:* [[Becoming_A_User| Becoming_A_User]]&amp;lt;br &amp;gt;&lt;br /&gt;
&lt;br /&gt;
Procedures&lt;br /&gt;
:* [[Transferring Files| Transferring Files]]&amp;lt;br &amp;gt;&lt;br /&gt;
:* [[Setting Up Your Environment| Setting Up Your Environment]]&amp;lt;br &amp;gt;&lt;br /&gt;
:* [[Login Nodes Versus Compute Nodes| Login Nodes Versus Compute Nodes]]&amp;lt;br &amp;gt;&lt;br /&gt;
:* [[Files &amp;amp; Storage| Files &amp;amp; Storage]]&amp;lt;br &amp;gt;&lt;br /&gt;
:* [[Backups| Backups]]&amp;lt;br &amp;gt;&lt;br /&gt;
:* [[Compiling Your Code| Compiling Your Code]]&amp;lt;br &amp;gt;&lt;br /&gt;
:* [[Running Your Code| Running Your Code]]&amp;lt;br &amp;gt;&lt;br /&gt;
:* [[Graphical Applications| Graphical Applications]]&amp;lt;br &amp;gt;&lt;br /&gt;
&lt;br /&gt;
TRACC Cluster Software&lt;br /&gt;
:* [[TRACC Cluster Software#Supported On ARROW| Supported On ARROW]]&amp;lt;br &amp;gt;&lt;br /&gt;
:* [[TRACC Cluster Software#Under Investigation| Under Investigation]]&amp;lt;br &amp;gt;&lt;br /&gt;
&lt;br /&gt;
TRACC Policies&lt;br /&gt;
:* [[TRACC Policies#Application Restrictions| Application Restrictions]]&amp;lt;br &amp;gt;&lt;br /&gt;
:* [[TRACC Policies#Data Retention| Data Retention]]&amp;lt;br &amp;gt;&lt;br /&gt;
:* [[TRACC Policies#Email| Email]]&amp;lt;br &amp;gt;&lt;br /&gt;
:* [[TRACC Policies#Login Nodes| Login Nodes]]&amp;lt;br &amp;gt;&lt;/div&gt;</summary>
		<author><name>Ley</name></author>
	</entry>
	<entry>
		<id>https://wiki.anl.gov/wiki_tracc/index.php?title=Main_Page&amp;diff=2478</id>
		<title>Main Page</title>
		<link rel="alternate" type="text/html" href="https://wiki.anl.gov/wiki_tracc/index.php?title=Main_Page&amp;diff=2478"/>
		<updated>2025-12-09T05:57:03Z</updated>

		<summary type="html">&lt;p&gt;Ley: /* LS-Dyna+ */&lt;/p&gt;
&lt;hr /&gt;
&lt;div&gt;&amp;lt;!--&lt;br /&gt;
&lt;br /&gt;
__NOTOC__&lt;br /&gt;
{| style=&amp;quot;width:100%; background:#7CA0C4; color:white&amp;quot; border=&amp;quot;0&amp;quot; cellpadding=&amp;quot;0&amp;quot; cellspacing=&amp;quot;0&amp;quot;&lt;br /&gt;
| align=&amp;quot;left&amp;quot; width=&amp;quot;99%&amp;quot;|&lt;br /&gt;
&amp;lt;font color=&amp;quot;white&amp;quot; size=&amp;quot;+1&amp;quot;&amp;gt;&amp;lt;blockquote&amp;gt;The Transportation Research and Analysis Computing Center (TRACC)&amp;lt;/blockquote&amp;gt;&amp;lt;/font&amp;gt;&amp;lt;p&amp;gt;&lt;br /&gt;
&lt;br /&gt;
&amp;lt;font color=&amp;quot;white&amp;quot;&amp;gt;&amp;lt;blockquote&amp;gt;&lt;br /&gt;
This TRACC wiki is an EXTERNAL Argonne collaboration site to support information exchange and collaboration among TRACC users, collaborators, and partners.  The purpose of this Wiki is to: &lt;br /&gt;
* Document the setup, operation, and FAQs about TRACC supported software and hardware resources. &lt;br /&gt;
* Maintain a repository for hardware and software improvements and modifications. &lt;br /&gt;
* Share TRACC technical staff, collaborator and user generated &amp;quot;How-To&amp;quot; procedures with the transportation research and development community.&lt;br /&gt;
| style=&amp;quot;width:1%&amp;quot; | [[Image:Best.JPG]]&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
--&amp;gt;&lt;br /&gt;
=The ARROW HPC Cluster at TRACC=&lt;br /&gt;
&amp;lt;I&amp;gt;Operated by Argonne National Laboratory for the U.S. Department of Transportation&amp;lt;/I&amp;gt;&lt;br /&gt;
==Introduction==&lt;br /&gt;
The ARROW HPC cluster at TRACC is being operated for the Federal Highway Administration (FHWA) and the National Highway Traffic Safety Administration (NTHSA) at the US Department of Transportation. The site operates at Argonne National Laboratory under the Transportation Research and Analysis Computing Center (TRACC). The site is funded by specific US Department of Transportation projects and is not available to the general research community.&lt;br /&gt;
&lt;br /&gt;
TRACC has combined the original hardware from the Phoenix and Zephyr clusters into a single HPC cluster called ARROW. This consolidation allows for the efficient administration of TRACC cluster services. To mitigate the problems with load balancing, the different types of hardware nodes on the ARROW cluster are partitioned into queues with specific characteristics. When new hardware is installed to expand cluster resources, it will be made typically available via a new queue.&lt;br /&gt;
&lt;br /&gt;
ARROW is arranged such that there is a set of four login nodes that are accessed by users using the SSH protocol. The login servers are configured identically so that it doesn&#039;t matter which login server a user is connected to (the caveat is that the user should connect to the same login server where he may have created a GUI desktop session). All servers and nodes share a single parallel file system, and an individual home directory is provided for each user to store their data.&lt;br /&gt;
&lt;br /&gt;
Some common software packages provided on the cluster are commercial engineering applications such as StarCCM+, and LS-Dyna. In addition, selected open source software is available, such as OpenFOAM. Specific versions of each software are selected by the user through the &amp;quot;module&amp;quot; command, which is commonly used on HPC clusters. Connections are limited to the SSH protocol at this time, and cyber security is a major aspect of our operation. We will work with you to optimize your remote operating experience. Users will be able to create and use GUI desktops using the XFCE software. These desktops will stay active until the users decides to log out. Otherwise, GUI sessions will stay active and can be reconnected to, similar to the Windows Remote Desktop software. Thus, users can connect and disconnect from these sessions at will, and network interruptions will typically not interfere with your work flow. The PBS job submission framework allows users to use HPC resources in an efficient manner, sharing a fairly complex selection of specialty systems. This includes typical HPC nodes, as well as selected GPU systems for AI/ML applications.&lt;br /&gt;
&lt;br /&gt;
The ARROW HPC cluster consists of about 200 servers and provides a substantial computing facility and research platform for USDOT, especially in the areas of Bridge and River Hydraulics as well as Vehicle Occupant Safety analysis. It has been optimized for HPC engineering applications in Computational Fluid Dynamics and Computational Structural Mechanics. &lt;br /&gt;
&amp;lt;!--&lt;br /&gt;
:* [[introduction | &#039;&#039;&#039;Introduction to the ARROW Cluster&#039;&#039;&#039;]]&amp;lt;br &amp;gt;&lt;br /&gt;
:* [[ARROW Cluster#ARROW Queues | &#039;&#039;&#039;ARROW Computing Queues&#039;&#039;&#039;]]&amp;lt;br &amp;gt;&lt;br /&gt;
:* [[Becoming A User| &#039;&#039;&#039;Becoming A User&#039;&#039;&#039;]]&amp;lt;br &amp;gt;&lt;br /&gt;
--&amp;gt;&lt;br /&gt;
&lt;br /&gt;
==User Accounts==&lt;br /&gt;
User accounts are granted to users identified by the U.S. Department of Transportation. The current sponsors for our computing facility are the Federal Highway Administration (FHWA) and the National Highway Traffic Safety Administration (NTHSA) at the US Department of Transportation. You may want to contact [mailto:TRACC_Director@anl.gov Dr. Hubert Ley] at Argonne National Laboratory to inquire about the potential support and use of the facility for one of your transportation research projects.&lt;br /&gt;
&lt;br /&gt;
If you are working on a project that is already sponsoring the facility, have your contact person at USDOT request an account for you. We will then send you a link to a form where you can enter all the relevant contact information, and we will then create the account for you. These accounts usually start with &amp;quot;ac.&amp;quot;, for example &amp;quot;ac.username&amp;quot;, and you will need to remember that when you log in to ARROW. We will communicate all details with you once the application progresses.&lt;br /&gt;
&lt;br /&gt;
==Using the Cluster==&lt;br /&gt;
&lt;br /&gt;
The following links can be used to find more detailed information on the various aspects of the ARROW cluster&#039;s design and operation. This includes information on how to access the cluster remotely, typical applications that can be used to access the cluster, the overall layout of working queues and the job submission procedure, aspects of running specific software applications, information about software compilations, selecting software modules, using graphical applications and user interfaces, information about redundancy and backups, and much more.&lt;br /&gt;
&lt;br /&gt;
:* [[How to Connect| How to Connect]]&amp;lt;br&amp;gt;&lt;br /&gt;
:* [[Transferring Files| Transferring Files]]&amp;lt;br&amp;gt;&lt;br /&gt;
:* [[Job Submission and Monitoring| Job Submission and Monitoring]]&amp;lt;br &amp;gt;&lt;br /&gt;
&lt;br /&gt;
==Software Application Documentation==&lt;br /&gt;
&lt;br /&gt;
The following links will not work when reading these page with an external browser. They are only available when viewing the pages within the Linux desktop GUI after being logged in.&lt;br /&gt;
&lt;br /&gt;
===StarCCM+===&lt;br /&gt;
&lt;br /&gt;
* Version 20.04.007-R8&lt;br /&gt;
** [http://146.139.58.10/starccm/20.04.007-R8/STAR-CCM+20.04.007-R8/doc/en/online/STARCCMP StarCCM+ Online User&#039;s Guide (English)]&lt;br /&gt;
** [http://146.139.58.10/starccm/20.04.007-R8/STAR-CCM+20.04.007-R8/doc/zh/online/STARCCMP StarCCM+ Online User&#039;s Guide (Chinese)]&lt;br /&gt;
** [http://146.139.58.10/starccm/20.04.007-R8/STAR-CCM+20.04.007-R8/doc/api/html StarCCM+ API Documentation]&lt;br /&gt;
** [http://146.139.58.10/starccm/20.04.007-R8/STAR-CCM+20.04.007-R8/doc/client/html StarCCM+ Java API Documentation]&lt;br /&gt;
** [http://146.139.58.10/starccm/20.04.007-R8/STAR-CCM+20.04.007-R8/doc/ StarCCM+ All Manuals and Release Notes]&lt;br /&gt;
&lt;br /&gt;
===LS-Dyna+===&lt;br /&gt;
&lt;br /&gt;
* LS-Dyna Version R16&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R16/LS-DYNA_Manual_Theory_R16.pdf LS-Dyna Theory Manual]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R16/LS-DYNA_Manual_Vol_I_R16.pdf LS-Dyna User&#039;s Manual]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R16/LS-DYNA_Manual_Vol_II_R16.pdf LS-Dyna User&#039;s Manual]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R16/LS-DYNA_Manual_Vol_III_R16.pdf LS-Dyna User&#039;s Manual]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R16/ All Manuals and Release Notes]&lt;br /&gt;
&lt;br /&gt;
* LS-Dyna Version R15&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R15/LS-DYNA_Manual_Theory_R15.pdf LS-Dyna Theory Manual]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R15/LS-DYNA_Manual_Volume_I_R15.pdf LS-Dyna User&#039;s Manual]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R15/LS-DYNA_Manual_Volume_II_R15.pdf LS-Dyna User&#039;s Manual]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R15/LS-DYNA_Manual_Volume_III_R15.pdf LS-Dyna User&#039;s Manual]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R15/ All Manuals and Release Notes]&lt;br /&gt;
&lt;br /&gt;
* LS-Dyna Version R14&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R14/LS-DYNA_Manual_Theory_R14.pdf LS-Dyna Theory Manual]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R14/LS-DYNA_Manual_Volume_I_R14.pdf LS-Dyna User&#039;s Manual]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R14/LS-DYNA_Manual_Volume_II_R14.pdf LS-Dyna User&#039;s Manual]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R14/LS-DYNA_Manual_Volume_III_R14.pdf LS-Dyna User&#039;s Manual]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R14/ All Manuals and Release Notes]&lt;br /&gt;
&lt;br /&gt;
* LS-Dyna Version R13&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R13/LS-DYNA_Manual_Theory_R13.pdf LS-Dyna Theory Manual]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R13/LS-DYNA_Manual_Volume_I_R13.pdf LS-Dyna User&#039;s Manual]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R13/LS-DYNA_Manual_Volume_II_R13.pdf LS-Dyna User&#039;s Manual]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R13/LS-DYNA_Manual_Volume_III_R13.pdf LS-Dyna User&#039;s Manual]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R13/ All Manuals and Release Notes]&lt;br /&gt;
&lt;br /&gt;
* LS-Dyna Version R12&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R12/LS-DYNA_Manual_Theory_R12.pdf LS-Dyna Theory Manual]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R12/LS-DYNA_Manual_Volume_I_R12.pdf LS-Dyna User&#039;s Manual]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R12/LS-DYNA_Manual_Volume_II_R12.pdf LS-Dyna User&#039;s Manual]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R12/LS-DYNA_Manual_Volume_III_R12.pdf LS-Dyna User&#039;s Manual]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R12/ All Manuals and Release Notes]&lt;br /&gt;
&lt;br /&gt;
* LS-Dyna Version R11&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R11/LS-DYNA_Manual_Theory_R11.pdf LS-Dyna Theory Manual]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R11/LS-DYNA_Manual_Volume_I_R11.pdf LS-Dyna User&#039;s Manual]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R11/LS-DYNA_Manual_Volume_II_R11.pdf LS-Dyna User&#039;s Manual]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R11/LS-DYNA_Manual_Volume_III_R11.pdf LS-Dyna User&#039;s Manual]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R11/ All Manuals and Release Notes]&lt;br /&gt;
&lt;br /&gt;
=The links and pages shown below are outdated!=&lt;br /&gt;
&lt;br /&gt;
&amp;lt;BLOCKQUOTE&amp;gt;&lt;br /&gt;
[[File:Attention.jpg|25px]] &#039;&#039;&#039;IMPORTANT NOTE:&#039;&#039;&#039; &#039;&#039;We are updating the wiki pages with the most recent information on services, hardware, and capabilities. The information provided below is outdated and in urgent need for a update. We recommend to disregard the information for now.&#039;&#039;&lt;br /&gt;
&amp;lt;/BLOCKQUOTE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
General Information&lt;br /&gt;
:* [[Becoming_A_User| Becoming_A_User]]&amp;lt;br &amp;gt;&lt;br /&gt;
&lt;br /&gt;
Procedures&lt;br /&gt;
:* [[Transferring Files| Transferring Files]]&amp;lt;br &amp;gt;&lt;br /&gt;
:* [[Setting Up Your Environment| Setting Up Your Environment]]&amp;lt;br &amp;gt;&lt;br /&gt;
:* [[Login Nodes Versus Compute Nodes| Login Nodes Versus Compute Nodes]]&amp;lt;br &amp;gt;&lt;br /&gt;
:* [[Files &amp;amp; Storage| Files &amp;amp; Storage]]&amp;lt;br &amp;gt;&lt;br /&gt;
:* [[Backups| Backups]]&amp;lt;br &amp;gt;&lt;br /&gt;
:* [[Compiling Your Code| Compiling Your Code]]&amp;lt;br &amp;gt;&lt;br /&gt;
:* [[Running Your Code| Running Your Code]]&amp;lt;br &amp;gt;&lt;br /&gt;
:* [[Graphical Applications| Graphical Applications]]&amp;lt;br &amp;gt;&lt;br /&gt;
&lt;br /&gt;
TRACC Cluster Software&lt;br /&gt;
:* [[TRACC Cluster Software#Supported On ARROW| Supported On ARROW]]&amp;lt;br &amp;gt;&lt;br /&gt;
:* [[TRACC Cluster Software#Under Investigation| Under Investigation]]&amp;lt;br &amp;gt;&lt;br /&gt;
&lt;br /&gt;
TRACC Policies&lt;br /&gt;
:* [[TRACC Policies#Application Restrictions| Application Restrictions]]&amp;lt;br &amp;gt;&lt;br /&gt;
:* [[TRACC Policies#Data Retention| Data Retention]]&amp;lt;br &amp;gt;&lt;br /&gt;
:* [[TRACC Policies#Email| Email]]&amp;lt;br &amp;gt;&lt;br /&gt;
:* [[TRACC Policies#Login Nodes| Login Nodes]]&amp;lt;br &amp;gt;&lt;/div&gt;</summary>
		<author><name>Ley</name></author>
	</entry>
	<entry>
		<id>https://wiki.anl.gov/wiki_tracc/index.php?title=Main_Page&amp;diff=2477</id>
		<title>Main Page</title>
		<link rel="alternate" type="text/html" href="https://wiki.anl.gov/wiki_tracc/index.php?title=Main_Page&amp;diff=2477"/>
		<updated>2025-12-09T05:54:40Z</updated>

		<summary type="html">&lt;p&gt;Ley: /* LS-Dyna+ */&lt;/p&gt;
&lt;hr /&gt;
&lt;div&gt;&amp;lt;!--&lt;br /&gt;
&lt;br /&gt;
__NOTOC__&lt;br /&gt;
{| style=&amp;quot;width:100%; background:#7CA0C4; color:white&amp;quot; border=&amp;quot;0&amp;quot; cellpadding=&amp;quot;0&amp;quot; cellspacing=&amp;quot;0&amp;quot;&lt;br /&gt;
| align=&amp;quot;left&amp;quot; width=&amp;quot;99%&amp;quot;|&lt;br /&gt;
&amp;lt;font color=&amp;quot;white&amp;quot; size=&amp;quot;+1&amp;quot;&amp;gt;&amp;lt;blockquote&amp;gt;The Transportation Research and Analysis Computing Center (TRACC)&amp;lt;/blockquote&amp;gt;&amp;lt;/font&amp;gt;&amp;lt;p&amp;gt;&lt;br /&gt;
&lt;br /&gt;
&amp;lt;font color=&amp;quot;white&amp;quot;&amp;gt;&amp;lt;blockquote&amp;gt;&lt;br /&gt;
This TRACC wiki is an EXTERNAL Argonne collaboration site to support information exchange and collaboration among TRACC users, collaborators, and partners.  The purpose of this Wiki is to: &lt;br /&gt;
* Document the setup, operation, and FAQs about TRACC supported software and hardware resources. &lt;br /&gt;
* Maintain a repository for hardware and software improvements and modifications. &lt;br /&gt;
* Share TRACC technical staff, collaborator and user generated &amp;quot;How-To&amp;quot; procedures with the transportation research and development community.&lt;br /&gt;
| style=&amp;quot;width:1%&amp;quot; | [[Image:Best.JPG]]&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
--&amp;gt;&lt;br /&gt;
=The ARROW HPC Cluster at TRACC=&lt;br /&gt;
&amp;lt;I&amp;gt;Operated by Argonne National Laboratory for the U.S. Department of Transportation&amp;lt;/I&amp;gt;&lt;br /&gt;
==Introduction==&lt;br /&gt;
The ARROW HPC cluster at TRACC is being operated for the Federal Highway Administration (FHWA) and the National Highway Traffic Safety Administration (NTHSA) at the US Department of Transportation. The site operates at Argonne National Laboratory under the Transportation Research and Analysis Computing Center (TRACC). The site is funded by specific US Department of Transportation projects and is not available to the general research community.&lt;br /&gt;
&lt;br /&gt;
TRACC has combined the original hardware from the Phoenix and Zephyr clusters into a single HPC cluster called ARROW. This consolidation allows for the efficient administration of TRACC cluster services. To mitigate the problems with load balancing, the different types of hardware nodes on the ARROW cluster are partitioned into queues with specific characteristics. When new hardware is installed to expand cluster resources, it will be made typically available via a new queue.&lt;br /&gt;
&lt;br /&gt;
ARROW is arranged such that there is a set of four login nodes that are accessed by users using the SSH protocol. The login servers are configured identically so that it doesn&#039;t matter which login server a user is connected to (the caveat is that the user should connect to the same login server where he may have created a GUI desktop session). All servers and nodes share a single parallel file system, and an individual home directory is provided for each user to store their data.&lt;br /&gt;
&lt;br /&gt;
Some common software packages provided on the cluster are commercial engineering applications such as StarCCM+, and LS-Dyna. In addition, selected open source software is available, such as OpenFOAM. Specific versions of each software are selected by the user through the &amp;quot;module&amp;quot; command, which is commonly used on HPC clusters. Connections are limited to the SSH protocol at this time, and cyber security is a major aspect of our operation. We will work with you to optimize your remote operating experience. Users will be able to create and use GUI desktops using the XFCE software. These desktops will stay active until the users decides to log out. Otherwise, GUI sessions will stay active and can be reconnected to, similar to the Windows Remote Desktop software. Thus, users can connect and disconnect from these sessions at will, and network interruptions will typically not interfere with your work flow. The PBS job submission framework allows users to use HPC resources in an efficient manner, sharing a fairly complex selection of specialty systems. This includes typical HPC nodes, as well as selected GPU systems for AI/ML applications.&lt;br /&gt;
&lt;br /&gt;
The ARROW HPC cluster consists of about 200 servers and provides a substantial computing facility and research platform for USDOT, especially in the areas of Bridge and River Hydraulics as well as Vehicle Occupant Safety analysis. It has been optimized for HPC engineering applications in Computational Fluid Dynamics and Computational Structural Mechanics. &lt;br /&gt;
&amp;lt;!--&lt;br /&gt;
:* [[introduction | &#039;&#039;&#039;Introduction to the ARROW Cluster&#039;&#039;&#039;]]&amp;lt;br &amp;gt;&lt;br /&gt;
:* [[ARROW Cluster#ARROW Queues | &#039;&#039;&#039;ARROW Computing Queues&#039;&#039;&#039;]]&amp;lt;br &amp;gt;&lt;br /&gt;
:* [[Becoming A User| &#039;&#039;&#039;Becoming A User&#039;&#039;&#039;]]&amp;lt;br &amp;gt;&lt;br /&gt;
--&amp;gt;&lt;br /&gt;
&lt;br /&gt;
==User Accounts==&lt;br /&gt;
User accounts are granted to users identified by the U.S. Department of Transportation. The current sponsors for our computing facility are the Federal Highway Administration (FHWA) and the National Highway Traffic Safety Administration (NTHSA) at the US Department of Transportation. You may want to contact [mailto:TRACC_Director@anl.gov Dr. Hubert Ley] at Argonne National Laboratory to inquire about the potential support and use of the facility for one of your transportation research projects.&lt;br /&gt;
&lt;br /&gt;
If you are working on a project that is already sponsoring the facility, have your contact person at USDOT request an account for you. We will then send you a link to a form where you can enter all the relevant contact information, and we will then create the account for you. These accounts usually start with &amp;quot;ac.&amp;quot;, for example &amp;quot;ac.username&amp;quot;, and you will need to remember that when you log in to ARROW. We will communicate all details with you once the application progresses.&lt;br /&gt;
&lt;br /&gt;
==Using the Cluster==&lt;br /&gt;
&lt;br /&gt;
The following links can be used to find more detailed information on the various aspects of the ARROW cluster&#039;s design and operation. This includes information on how to access the cluster remotely, typical applications that can be used to access the cluster, the overall layout of working queues and the job submission procedure, aspects of running specific software applications, information about software compilations, selecting software modules, using graphical applications and user interfaces, information about redundancy and backups, and much more.&lt;br /&gt;
&lt;br /&gt;
:* [[How to Connect| How to Connect]]&amp;lt;br&amp;gt;&lt;br /&gt;
:* [[Transferring Files| Transferring Files]]&amp;lt;br&amp;gt;&lt;br /&gt;
:* [[Job Submission and Monitoring| Job Submission and Monitoring]]&amp;lt;br &amp;gt;&lt;br /&gt;
&lt;br /&gt;
==Software Application Documentation==&lt;br /&gt;
&lt;br /&gt;
The following links will not work when reading these page with an external browser. They are only available when viewing the pages within the Linux desktop GUI after being logged in.&lt;br /&gt;
&lt;br /&gt;
===StarCCM+===&lt;br /&gt;
&lt;br /&gt;
* Version 20.04.007-R8&lt;br /&gt;
** [http://146.139.58.10/starccm/20.04.007-R8/STAR-CCM+20.04.007-R8/doc/en/online/STARCCMP StarCCM+ Online User&#039;s Guide (English)]&lt;br /&gt;
** [http://146.139.58.10/starccm/20.04.007-R8/STAR-CCM+20.04.007-R8/doc/zh/online/STARCCMP StarCCM+ Online User&#039;s Guide (Chinese)]&lt;br /&gt;
** [http://146.139.58.10/starccm/20.04.007-R8/STAR-CCM+20.04.007-R8/doc/api/html StarCCM+ API Documentation]&lt;br /&gt;
** [http://146.139.58.10/starccm/20.04.007-R8/STAR-CCM+20.04.007-R8/doc/client/html StarCCM+ Java API Documentation]&lt;br /&gt;
** [http://146.139.58.10/starccm/20.04.007-R8/STAR-CCM+20.04.007-R8/doc/ StarCCM+ All Manuals and Release Notes]&lt;br /&gt;
&lt;br /&gt;
===LS-Dyna+===&lt;br /&gt;
&lt;br /&gt;
* LS-Dyna Version R16&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R16/LS-DYNA_Manual_Theory_R16.pdf LS-Dyna Theory Manual]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R16/LS-DYNA_Manual_Vol_I_R16.pdf LS-Dyna User&#039;s Manual]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R16/LS-DYNA_Manual_Vol_II_R16.pdf LS-Dyna User&#039;s Manual]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R16/LS-DYNA_Manual_Vol_III_R16.pdf LS-Dyna User&#039;s Manual]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R16/ All Manuals and Release Notes]&lt;br /&gt;
&lt;br /&gt;
* LS-Dyna Version R15&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R15/LS-DYNA_Manual_Theory_R15.pdf LS-Dyna Theory Manual]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R15/LS-DYNA_Manual_Vol_I_R15.pdf LS-Dyna User&#039;s Manual]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R15/LS-DYNA_Manual_Vol_II_R15.pdf LS-Dyna User&#039;s Manual]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R15/LS-DYNA_Manual_Vol_III_R15.pdf LS-Dyna User&#039;s Manual]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R15/ All Manuals and Release Notes]&lt;br /&gt;
&lt;br /&gt;
* LS-Dyna Version R14&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R14/LS-DYNA_Manual_Theory_R14.pdf LS-Dyna Theory Manual]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R14/LS-DYNA_Manual_Vol_I_R14.pdf LS-Dyna User&#039;s Manual]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R14/LS-DYNA_Manual_Vol_II_R14.pdf LS-Dyna User&#039;s Manual]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R14/LS-DYNA_Manual_Vol_III_R14.pdf LS-Dyna User&#039;s Manual]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R14/ All Manuals and Release Notes]&lt;br /&gt;
&lt;br /&gt;
* LS-Dyna Version R13&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R13/LS-DYNA_Manual_Theory_R13.pdf LS-Dyna Theory Manual]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R13/LS-DYNA_Manual_Vol_I_R13.pdf LS-Dyna User&#039;s Manual]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R13/LS-DYNA_Manual_Vol_II_R13.pdf LS-Dyna User&#039;s Manual]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R13/LS-DYNA_Manual_Vol_III_R13.pdf LS-Dyna User&#039;s Manual]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R13/ All Manuals and Release Notes]&lt;br /&gt;
&lt;br /&gt;
* LS-Dyna Version R12&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R12/LS-DYNA_Manual_Theory_R12.pdf LS-Dyna Theory Manual]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R12/LS-DYNA_Manual_Vol_I_R12.pdf LS-Dyna User&#039;s Manual]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R12/LS-DYNA_Manual_Vol_II_R12.pdf LS-Dyna User&#039;s Manual]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R12/LS-DYNA_Manual_Vol_III_R12.pdf LS-Dyna User&#039;s Manual]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R12/ All Manuals and Release Notes]&lt;br /&gt;
&lt;br /&gt;
* LS-Dyna Version R11&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R11/LS-DYNA_Manual_Theory_R11.pdf LS-Dyna Theory Manual]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R11/LS-DYNA_Manual_Vol_I_R11.pdf LS-Dyna User&#039;s Manual]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R11/LS-DYNA_Manual_Vol_II_R11.pdf LS-Dyna User&#039;s Manual]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R11/LS-DYNA_Manual_Vol_III_R11.pdf LS-Dyna User&#039;s Manual]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R11/ All Manuals and Release Notes]&lt;br /&gt;
&lt;br /&gt;
=The links and pages shown below are outdated!=&lt;br /&gt;
&lt;br /&gt;
&amp;lt;BLOCKQUOTE&amp;gt;&lt;br /&gt;
[[File:Attention.jpg|25px]] &#039;&#039;&#039;IMPORTANT NOTE:&#039;&#039;&#039; &#039;&#039;We are updating the wiki pages with the most recent information on services, hardware, and capabilities. The information provided below is outdated and in urgent need for a update. We recommend to disregard the information for now.&#039;&#039;&lt;br /&gt;
&amp;lt;/BLOCKQUOTE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
General Information&lt;br /&gt;
:* [[Becoming_A_User| Becoming_A_User]]&amp;lt;br &amp;gt;&lt;br /&gt;
&lt;br /&gt;
Procedures&lt;br /&gt;
:* [[Transferring Files| Transferring Files]]&amp;lt;br &amp;gt;&lt;br /&gt;
:* [[Setting Up Your Environment| Setting Up Your Environment]]&amp;lt;br &amp;gt;&lt;br /&gt;
:* [[Login Nodes Versus Compute Nodes| Login Nodes Versus Compute Nodes]]&amp;lt;br &amp;gt;&lt;br /&gt;
:* [[Files &amp;amp; Storage| Files &amp;amp; Storage]]&amp;lt;br &amp;gt;&lt;br /&gt;
:* [[Backups| Backups]]&amp;lt;br &amp;gt;&lt;br /&gt;
:* [[Compiling Your Code| Compiling Your Code]]&amp;lt;br &amp;gt;&lt;br /&gt;
:* [[Running Your Code| Running Your Code]]&amp;lt;br &amp;gt;&lt;br /&gt;
:* [[Graphical Applications| Graphical Applications]]&amp;lt;br &amp;gt;&lt;br /&gt;
&lt;br /&gt;
TRACC Cluster Software&lt;br /&gt;
:* [[TRACC Cluster Software#Supported On ARROW| Supported On ARROW]]&amp;lt;br &amp;gt;&lt;br /&gt;
:* [[TRACC Cluster Software#Under Investigation| Under Investigation]]&amp;lt;br &amp;gt;&lt;br /&gt;
&lt;br /&gt;
TRACC Policies&lt;br /&gt;
:* [[TRACC Policies#Application Restrictions| Application Restrictions]]&amp;lt;br &amp;gt;&lt;br /&gt;
:* [[TRACC Policies#Data Retention| Data Retention]]&amp;lt;br &amp;gt;&lt;br /&gt;
:* [[TRACC Policies#Email| Email]]&amp;lt;br &amp;gt;&lt;br /&gt;
:* [[TRACC Policies#Login Nodes| Login Nodes]]&amp;lt;br &amp;gt;&lt;/div&gt;</summary>
		<author><name>Ley</name></author>
	</entry>
	<entry>
		<id>https://wiki.anl.gov/wiki_tracc/index.php?title=Main_Page&amp;diff=2476</id>
		<title>Main Page</title>
		<link rel="alternate" type="text/html" href="https://wiki.anl.gov/wiki_tracc/index.php?title=Main_Page&amp;diff=2476"/>
		<updated>2025-12-09T05:52:52Z</updated>

		<summary type="html">&lt;p&gt;Ley: /* LS-Dyna+ */&lt;/p&gt;
&lt;hr /&gt;
&lt;div&gt;&amp;lt;!--&lt;br /&gt;
&lt;br /&gt;
__NOTOC__&lt;br /&gt;
{| style=&amp;quot;width:100%; background:#7CA0C4; color:white&amp;quot; border=&amp;quot;0&amp;quot; cellpadding=&amp;quot;0&amp;quot; cellspacing=&amp;quot;0&amp;quot;&lt;br /&gt;
| align=&amp;quot;left&amp;quot; width=&amp;quot;99%&amp;quot;|&lt;br /&gt;
&amp;lt;font color=&amp;quot;white&amp;quot; size=&amp;quot;+1&amp;quot;&amp;gt;&amp;lt;blockquote&amp;gt;The Transportation Research and Analysis Computing Center (TRACC)&amp;lt;/blockquote&amp;gt;&amp;lt;/font&amp;gt;&amp;lt;p&amp;gt;&lt;br /&gt;
&lt;br /&gt;
&amp;lt;font color=&amp;quot;white&amp;quot;&amp;gt;&amp;lt;blockquote&amp;gt;&lt;br /&gt;
This TRACC wiki is an EXTERNAL Argonne collaboration site to support information exchange and collaboration among TRACC users, collaborators, and partners.  The purpose of this Wiki is to: &lt;br /&gt;
* Document the setup, operation, and FAQs about TRACC supported software and hardware resources. &lt;br /&gt;
* Maintain a repository for hardware and software improvements and modifications. &lt;br /&gt;
* Share TRACC technical staff, collaborator and user generated &amp;quot;How-To&amp;quot; procedures with the transportation research and development community.&lt;br /&gt;
| style=&amp;quot;width:1%&amp;quot; | [[Image:Best.JPG]]&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
--&amp;gt;&lt;br /&gt;
=The ARROW HPC Cluster at TRACC=&lt;br /&gt;
&amp;lt;I&amp;gt;Operated by Argonne National Laboratory for the U.S. Department of Transportation&amp;lt;/I&amp;gt;&lt;br /&gt;
==Introduction==&lt;br /&gt;
The ARROW HPC cluster at TRACC is being operated for the Federal Highway Administration (FHWA) and the National Highway Traffic Safety Administration (NTHSA) at the US Department of Transportation. The site operates at Argonne National Laboratory under the Transportation Research and Analysis Computing Center (TRACC). The site is funded by specific US Department of Transportation projects and is not available to the general research community.&lt;br /&gt;
&lt;br /&gt;
TRACC has combined the original hardware from the Phoenix and Zephyr clusters into a single HPC cluster called ARROW. This consolidation allows for the efficient administration of TRACC cluster services. To mitigate the problems with load balancing, the different types of hardware nodes on the ARROW cluster are partitioned into queues with specific characteristics. When new hardware is installed to expand cluster resources, it will be made typically available via a new queue.&lt;br /&gt;
&lt;br /&gt;
ARROW is arranged such that there is a set of four login nodes that are accessed by users using the SSH protocol. The login servers are configured identically so that it doesn&#039;t matter which login server a user is connected to (the caveat is that the user should connect to the same login server where he may have created a GUI desktop session). All servers and nodes share a single parallel file system, and an individual home directory is provided for each user to store their data.&lt;br /&gt;
&lt;br /&gt;
Some common software packages provided on the cluster are commercial engineering applications such as StarCCM+, and LS-Dyna. In addition, selected open source software is available, such as OpenFOAM. Specific versions of each software are selected by the user through the &amp;quot;module&amp;quot; command, which is commonly used on HPC clusters. Connections are limited to the SSH protocol at this time, and cyber security is a major aspect of our operation. We will work with you to optimize your remote operating experience. Users will be able to create and use GUI desktops using the XFCE software. These desktops will stay active until the users decides to log out. Otherwise, GUI sessions will stay active and can be reconnected to, similar to the Windows Remote Desktop software. Thus, users can connect and disconnect from these sessions at will, and network interruptions will typically not interfere with your work flow. The PBS job submission framework allows users to use HPC resources in an efficient manner, sharing a fairly complex selection of specialty systems. This includes typical HPC nodes, as well as selected GPU systems for AI/ML applications.&lt;br /&gt;
&lt;br /&gt;
The ARROW HPC cluster consists of about 200 servers and provides a substantial computing facility and research platform for USDOT, especially in the areas of Bridge and River Hydraulics as well as Vehicle Occupant Safety analysis. It has been optimized for HPC engineering applications in Computational Fluid Dynamics and Computational Structural Mechanics. &lt;br /&gt;
&amp;lt;!--&lt;br /&gt;
:* [[introduction | &#039;&#039;&#039;Introduction to the ARROW Cluster&#039;&#039;&#039;]]&amp;lt;br &amp;gt;&lt;br /&gt;
:* [[ARROW Cluster#ARROW Queues | &#039;&#039;&#039;ARROW Computing Queues&#039;&#039;&#039;]]&amp;lt;br &amp;gt;&lt;br /&gt;
:* [[Becoming A User| &#039;&#039;&#039;Becoming A User&#039;&#039;&#039;]]&amp;lt;br &amp;gt;&lt;br /&gt;
--&amp;gt;&lt;br /&gt;
&lt;br /&gt;
==User Accounts==&lt;br /&gt;
User accounts are granted to users identified by the U.S. Department of Transportation. The current sponsors for our computing facility are the Federal Highway Administration (FHWA) and the National Highway Traffic Safety Administration (NTHSA) at the US Department of Transportation. You may want to contact [mailto:TRACC_Director@anl.gov Dr. Hubert Ley] at Argonne National Laboratory to inquire about the potential support and use of the facility for one of your transportation research projects.&lt;br /&gt;
&lt;br /&gt;
If you are working on a project that is already sponsoring the facility, have your contact person at USDOT request an account for you. We will then send you a link to a form where you can enter all the relevant contact information, and we will then create the account for you. These accounts usually start with &amp;quot;ac.&amp;quot;, for example &amp;quot;ac.username&amp;quot;, and you will need to remember that when you log in to ARROW. We will communicate all details with you once the application progresses.&lt;br /&gt;
&lt;br /&gt;
==Using the Cluster==&lt;br /&gt;
&lt;br /&gt;
The following links can be used to find more detailed information on the various aspects of the ARROW cluster&#039;s design and operation. This includes information on how to access the cluster remotely, typical applications that can be used to access the cluster, the overall layout of working queues and the job submission procedure, aspects of running specific software applications, information about software compilations, selecting software modules, using graphical applications and user interfaces, information about redundancy and backups, and much more.&lt;br /&gt;
&lt;br /&gt;
:* [[How to Connect| How to Connect]]&amp;lt;br&amp;gt;&lt;br /&gt;
:* [[Transferring Files| Transferring Files]]&amp;lt;br&amp;gt;&lt;br /&gt;
:* [[Job Submission and Monitoring| Job Submission and Monitoring]]&amp;lt;br &amp;gt;&lt;br /&gt;
&lt;br /&gt;
==Software Application Documentation==&lt;br /&gt;
&lt;br /&gt;
The following links will not work when reading these page with an external browser. They are only available when viewing the pages within the Linux desktop GUI after being logged in.&lt;br /&gt;
&lt;br /&gt;
===StarCCM+===&lt;br /&gt;
&lt;br /&gt;
* Version 20.04.007-R8&lt;br /&gt;
** [http://146.139.58.10/starccm/20.04.007-R8/STAR-CCM+20.04.007-R8/doc/en/online/STARCCMP StarCCM+ Online User&#039;s Guide (English)]&lt;br /&gt;
** [http://146.139.58.10/starccm/20.04.007-R8/STAR-CCM+20.04.007-R8/doc/zh/online/STARCCMP StarCCM+ Online User&#039;s Guide (Chinese)]&lt;br /&gt;
** [http://146.139.58.10/starccm/20.04.007-R8/STAR-CCM+20.04.007-R8/doc/api/html StarCCM+ API Documentation]&lt;br /&gt;
** [http://146.139.58.10/starccm/20.04.007-R8/STAR-CCM+20.04.007-R8/doc/client/html StarCCM+ Java API Documentation]&lt;br /&gt;
** [http://146.139.58.10/starccm/20.04.007-R8/STAR-CCM+20.04.007-R8/doc/ StarCCM+ All Manuals and Release Notes]&lt;br /&gt;
&lt;br /&gt;
===LS-Dyna+===&lt;br /&gt;
&lt;br /&gt;
* LS-Dyna Version 16&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R16/LS-DYNA_Manual_Theory_R16.pdf LS-Dyna Theory Manual]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R16/LS-DYNA_Manual_Vol_I_R16.pdf LS-Dyna User&#039;s Manual]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R16/LS-DYNA_Manual_Vol_II_R16.pdf LS-Dyna User&#039;s Manual]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R16/LS-DYNA_Manual_Vol_III_R16.pdf LS-Dyna User&#039;s Manual]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R16/ All Manuals and Release Notes]&lt;br /&gt;
&lt;br /&gt;
* LS-Dyna Version 16&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R15/LS-DYNA_Manual_Theory_R15.pdf LS-Dyna Theory Manual]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R15/LS-DYNA_Manual_Vol_I_R15.pdf LS-Dyna User&#039;s Manual]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R15/LS-DYNA_Manual_Vol_II_R15.pdf LS-Dyna User&#039;s Manual]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R15/LS-DYNA_Manual_Vol_III_R15.pdf LS-Dyna User&#039;s Manual]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R15/ All Manuals and Release Notes]&lt;br /&gt;
&lt;br /&gt;
* LS-Dyna Version 16&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R14/LS-DYNA_Manual_Theory_R14.pdf LS-Dyna Theory Manual]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R14/LS-DYNA_Manual_Vol_I_R14.pdf LS-Dyna User&#039;s Manual]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R14/LS-DYNA_Manual_Vol_II_R14.pdf LS-Dyna User&#039;s Manual]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R14/LS-DYNA_Manual_Vol_III_R14.pdf LS-Dyna User&#039;s Manual]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R14/ All Manuals and Release Notes]&lt;br /&gt;
&lt;br /&gt;
* LS-Dyna Version 16&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R13/LS-DYNA_Manual_Theory_R13.pdf LS-Dyna Theory Manual]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R13/LS-DYNA_Manual_Vol_I_R13.pdf LS-Dyna User&#039;s Manual]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R13/LS-DYNA_Manual_Vol_II_R13.pdf LS-Dyna User&#039;s Manual]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R13/LS-DYNA_Manual_Vol_III_R13.pdf LS-Dyna User&#039;s Manual]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R13/ All Manuals and Release Notes]&lt;br /&gt;
&lt;br /&gt;
* LS-Dyna Version 16&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R12/LS-DYNA_Manual_Theory_R12.pdf LS-Dyna Theory Manual]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R12/LS-DYNA_Manual_Vol_I_R12.pdf LS-Dyna User&#039;s Manual]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R12/LS-DYNA_Manual_Vol_II_R12.pdf LS-Dyna User&#039;s Manual]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R12/LS-DYNA_Manual_Vol_III_R12.pdf LS-Dyna User&#039;s Manual]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R12/ All Manuals and Release Notes]&lt;br /&gt;
&lt;br /&gt;
* LS-Dyna Version 16&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R11/LS-DYNA_Manual_Theory_R11.pdf LS-Dyna Theory Manual]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R11/LS-DYNA_Manual_Vol_I_R11.pdf LS-Dyna User&#039;s Manual]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R11/LS-DYNA_Manual_Vol_II_R11.pdf LS-Dyna User&#039;s Manual]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R11/LS-DYNA_Manual_Vol_III_R11.pdf LS-Dyna User&#039;s Manual]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R11/ All Manuals and Release Notes]&lt;br /&gt;
&lt;br /&gt;
=The links and pages shown below are outdated!=&lt;br /&gt;
&lt;br /&gt;
&amp;lt;BLOCKQUOTE&amp;gt;&lt;br /&gt;
[[File:Attention.jpg|25px]] &#039;&#039;&#039;IMPORTANT NOTE:&#039;&#039;&#039; &#039;&#039;We are updating the wiki pages with the most recent information on services, hardware, and capabilities. The information provided below is outdated and in urgent need for a update. We recommend to disregard the information for now.&#039;&#039;&lt;br /&gt;
&amp;lt;/BLOCKQUOTE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
General Information&lt;br /&gt;
:* [[Becoming_A_User| Becoming_A_User]]&amp;lt;br &amp;gt;&lt;br /&gt;
&lt;br /&gt;
Procedures&lt;br /&gt;
:* [[Transferring Files| Transferring Files]]&amp;lt;br &amp;gt;&lt;br /&gt;
:* [[Setting Up Your Environment| Setting Up Your Environment]]&amp;lt;br &amp;gt;&lt;br /&gt;
:* [[Login Nodes Versus Compute Nodes| Login Nodes Versus Compute Nodes]]&amp;lt;br &amp;gt;&lt;br /&gt;
:* [[Files &amp;amp; Storage| Files &amp;amp; Storage]]&amp;lt;br &amp;gt;&lt;br /&gt;
:* [[Backups| Backups]]&amp;lt;br &amp;gt;&lt;br /&gt;
:* [[Compiling Your Code| Compiling Your Code]]&amp;lt;br &amp;gt;&lt;br /&gt;
:* [[Running Your Code| Running Your Code]]&amp;lt;br &amp;gt;&lt;br /&gt;
:* [[Graphical Applications| Graphical Applications]]&amp;lt;br &amp;gt;&lt;br /&gt;
&lt;br /&gt;
TRACC Cluster Software&lt;br /&gt;
:* [[TRACC Cluster Software#Supported On ARROW| Supported On ARROW]]&amp;lt;br &amp;gt;&lt;br /&gt;
:* [[TRACC Cluster Software#Under Investigation| Under Investigation]]&amp;lt;br &amp;gt;&lt;br /&gt;
&lt;br /&gt;
TRACC Policies&lt;br /&gt;
:* [[TRACC Policies#Application Restrictions| Application Restrictions]]&amp;lt;br &amp;gt;&lt;br /&gt;
:* [[TRACC Policies#Data Retention| Data Retention]]&amp;lt;br &amp;gt;&lt;br /&gt;
:* [[TRACC Policies#Email| Email]]&amp;lt;br &amp;gt;&lt;br /&gt;
:* [[TRACC Policies#Login Nodes| Login Nodes]]&amp;lt;br &amp;gt;&lt;/div&gt;</summary>
		<author><name>Ley</name></author>
	</entry>
	<entry>
		<id>https://wiki.anl.gov/wiki_tracc/index.php?title=Main_Page&amp;diff=2475</id>
		<title>Main Page</title>
		<link rel="alternate" type="text/html" href="https://wiki.anl.gov/wiki_tracc/index.php?title=Main_Page&amp;diff=2475"/>
		<updated>2025-12-09T05:49:56Z</updated>

		<summary type="html">&lt;p&gt;Ley: /* StarCCM+ */&lt;/p&gt;
&lt;hr /&gt;
&lt;div&gt;&amp;lt;!--&lt;br /&gt;
&lt;br /&gt;
__NOTOC__&lt;br /&gt;
{| style=&amp;quot;width:100%; background:#7CA0C4; color:white&amp;quot; border=&amp;quot;0&amp;quot; cellpadding=&amp;quot;0&amp;quot; cellspacing=&amp;quot;0&amp;quot;&lt;br /&gt;
| align=&amp;quot;left&amp;quot; width=&amp;quot;99%&amp;quot;|&lt;br /&gt;
&amp;lt;font color=&amp;quot;white&amp;quot; size=&amp;quot;+1&amp;quot;&amp;gt;&amp;lt;blockquote&amp;gt;The Transportation Research and Analysis Computing Center (TRACC)&amp;lt;/blockquote&amp;gt;&amp;lt;/font&amp;gt;&amp;lt;p&amp;gt;&lt;br /&gt;
&lt;br /&gt;
&amp;lt;font color=&amp;quot;white&amp;quot;&amp;gt;&amp;lt;blockquote&amp;gt;&lt;br /&gt;
This TRACC wiki is an EXTERNAL Argonne collaboration site to support information exchange and collaboration among TRACC users, collaborators, and partners.  The purpose of this Wiki is to: &lt;br /&gt;
* Document the setup, operation, and FAQs about TRACC supported software and hardware resources. &lt;br /&gt;
* Maintain a repository for hardware and software improvements and modifications. &lt;br /&gt;
* Share TRACC technical staff, collaborator and user generated &amp;quot;How-To&amp;quot; procedures with the transportation research and development community.&lt;br /&gt;
| style=&amp;quot;width:1%&amp;quot; | [[Image:Best.JPG]]&lt;br /&gt;
|}&lt;br /&gt;
&lt;br /&gt;
--&amp;gt;&lt;br /&gt;
=The ARROW HPC Cluster at TRACC=&lt;br /&gt;
&amp;lt;I&amp;gt;Operated by Argonne National Laboratory for the U.S. Department of Transportation&amp;lt;/I&amp;gt;&lt;br /&gt;
==Introduction==&lt;br /&gt;
The ARROW HPC cluster at TRACC is being operated for the Federal Highway Administration (FHWA) and the National Highway Traffic Safety Administration (NTHSA) at the US Department of Transportation. The site operates at Argonne National Laboratory under the Transportation Research and Analysis Computing Center (TRACC). The site is funded by specific US Department of Transportation projects and is not available to the general research community.&lt;br /&gt;
&lt;br /&gt;
TRACC has combined the original hardware from the Phoenix and Zephyr clusters into a single HPC cluster called ARROW. This consolidation allows for the efficient administration of TRACC cluster services. To mitigate the problems with load balancing, the different types of hardware nodes on the ARROW cluster are partitioned into queues with specific characteristics. When new hardware is installed to expand cluster resources, it will be made typically available via a new queue.&lt;br /&gt;
&lt;br /&gt;
ARROW is arranged such that there is a set of four login nodes that are accessed by users using the SSH protocol. The login servers are configured identically so that it doesn&#039;t matter which login server a user is connected to (the caveat is that the user should connect to the same login server where he may have created a GUI desktop session). All servers and nodes share a single parallel file system, and an individual home directory is provided for each user to store their data.&lt;br /&gt;
&lt;br /&gt;
Some common software packages provided on the cluster are commercial engineering applications such as StarCCM+, and LS-Dyna. In addition, selected open source software is available, such as OpenFOAM. Specific versions of each software are selected by the user through the &amp;quot;module&amp;quot; command, which is commonly used on HPC clusters. Connections are limited to the SSH protocol at this time, and cyber security is a major aspect of our operation. We will work with you to optimize your remote operating experience. Users will be able to create and use GUI desktops using the XFCE software. These desktops will stay active until the users decides to log out. Otherwise, GUI sessions will stay active and can be reconnected to, similar to the Windows Remote Desktop software. Thus, users can connect and disconnect from these sessions at will, and network interruptions will typically not interfere with your work flow. The PBS job submission framework allows users to use HPC resources in an efficient manner, sharing a fairly complex selection of specialty systems. This includes typical HPC nodes, as well as selected GPU systems for AI/ML applications.&lt;br /&gt;
&lt;br /&gt;
The ARROW HPC cluster consists of about 200 servers and provides a substantial computing facility and research platform for USDOT, especially in the areas of Bridge and River Hydraulics as well as Vehicle Occupant Safety analysis. It has been optimized for HPC engineering applications in Computational Fluid Dynamics and Computational Structural Mechanics. &lt;br /&gt;
&amp;lt;!--&lt;br /&gt;
:* [[introduction | &#039;&#039;&#039;Introduction to the ARROW Cluster&#039;&#039;&#039;]]&amp;lt;br &amp;gt;&lt;br /&gt;
:* [[ARROW Cluster#ARROW Queues | &#039;&#039;&#039;ARROW Computing Queues&#039;&#039;&#039;]]&amp;lt;br &amp;gt;&lt;br /&gt;
:* [[Becoming A User| &#039;&#039;&#039;Becoming A User&#039;&#039;&#039;]]&amp;lt;br &amp;gt;&lt;br /&gt;
--&amp;gt;&lt;br /&gt;
&lt;br /&gt;
==User Accounts==&lt;br /&gt;
User accounts are granted to users identified by the U.S. Department of Transportation. The current sponsors for our computing facility are the Federal Highway Administration (FHWA) and the National Highway Traffic Safety Administration (NTHSA) at the US Department of Transportation. You may want to contact [mailto:TRACC_Director@anl.gov Dr. Hubert Ley] at Argonne National Laboratory to inquire about the potential support and use of the facility for one of your transportation research projects.&lt;br /&gt;
&lt;br /&gt;
If you are working on a project that is already sponsoring the facility, have your contact person at USDOT request an account for you. We will then send you a link to a form where you can enter all the relevant contact information, and we will then create the account for you. These accounts usually start with &amp;quot;ac.&amp;quot;, for example &amp;quot;ac.username&amp;quot;, and you will need to remember that when you log in to ARROW. We will communicate all details with you once the application progresses.&lt;br /&gt;
&lt;br /&gt;
==Using the Cluster==&lt;br /&gt;
&lt;br /&gt;
The following links can be used to find more detailed information on the various aspects of the ARROW cluster&#039;s design and operation. This includes information on how to access the cluster remotely, typical applications that can be used to access the cluster, the overall layout of working queues and the job submission procedure, aspects of running specific software applications, information about software compilations, selecting software modules, using graphical applications and user interfaces, information about redundancy and backups, and much more.&lt;br /&gt;
&lt;br /&gt;
:* [[How to Connect| How to Connect]]&amp;lt;br&amp;gt;&lt;br /&gt;
:* [[Transferring Files| Transferring Files]]&amp;lt;br&amp;gt;&lt;br /&gt;
:* [[Job Submission and Monitoring| Job Submission and Monitoring]]&amp;lt;br &amp;gt;&lt;br /&gt;
&lt;br /&gt;
==Software Application Documentation==&lt;br /&gt;
&lt;br /&gt;
The following links will not work when reading these page with an external browser. They are only available when viewing the pages within the Linux desktop GUI after being logged in.&lt;br /&gt;
&lt;br /&gt;
===StarCCM+===&lt;br /&gt;
&lt;br /&gt;
* Version 20.04.007-R8&lt;br /&gt;
** [http://146.139.58.10/starccm/20.04.007-R8/STAR-CCM+20.04.007-R8/doc/en/online/STARCCMP StarCCM+ Online User&#039;s Guide (English)]&lt;br /&gt;
** [http://146.139.58.10/starccm/20.04.007-R8/STAR-CCM+20.04.007-R8/doc/zh/online/STARCCMP StarCCM+ Online User&#039;s Guide (Chinese)]&lt;br /&gt;
** [http://146.139.58.10/starccm/20.04.007-R8/STAR-CCM+20.04.007-R8/doc/api/html StarCCM+ API Documentation]&lt;br /&gt;
** [http://146.139.58.10/starccm/20.04.007-R8/STAR-CCM+20.04.007-R8/doc/client/html StarCCM+ Java API Documentation]&lt;br /&gt;
** [http://146.139.58.10/starccm/20.04.007-R8/STAR-CCM+20.04.007-R8/doc/ StarCCM+ All Manuals and Release Notes]&lt;br /&gt;
&lt;br /&gt;
===LS-Dyna+===&lt;br /&gt;
&lt;br /&gt;
* LS-Dyna Version 16&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R16/LS-DYNA_Manual_Theory_R16.pdf LS-Dyna Theory Manual]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R16/LS-DYNA_Manual_Vol_I_R16.pdf LS-Dyna User&#039;s Manual]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R16/LS-DYNA_Manual_Vol_II_R16.pdf LS-Dyna User&#039;s Manual]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R16/LS-DYNA_Manual_Vol_III_R16.pdf LS-Dyna User&#039;s Manual]&lt;br /&gt;
** [http://146.139.58.10/lsdyna/Manuals_R16/ All Manuals and Release Notes]&lt;br /&gt;
&lt;br /&gt;
=The links and pages shown below are outdated!=&lt;br /&gt;
&lt;br /&gt;
&amp;lt;BLOCKQUOTE&amp;gt;&lt;br /&gt;
[[File:Attention.jpg|25px]] &#039;&#039;&#039;IMPORTANT NOTE:&#039;&#039;&#039; &#039;&#039;We are updating the wiki pages with the most recent information on services, hardware, and capabilities. The information provided below is outdated and in urgent need for a update. We recommend to disregard the information for now.&#039;&#039;&lt;br /&gt;
&amp;lt;/BLOCKQUOTE&amp;gt;&lt;br /&gt;
&lt;br /&gt;
General Information&lt;br /&gt;
:* [[Becoming_A_User| Becoming_A_User]]&amp;lt;br &amp;gt;&lt;br /&gt;
&lt;br /&gt;
Procedures&lt;br /&gt;
:* [[Transferring Files| Transferring Files]]&amp;lt;br &amp;gt;&lt;br /&gt;
:* [[Setting Up Your Environment| Setting Up Your Environment]]&amp;lt;br &amp;gt;&lt;br /&gt;
:* [[Login Nodes Versus Compute Nodes| Login Nodes Versus Compute Nodes]]&amp;lt;br &amp;gt;&lt;br /&gt;
:* [[Files &amp;amp; Storage| Files &amp;amp; Storage]]&amp;lt;br &amp;gt;&lt;br /&gt;
:* [[Backups| Backups]]&amp;lt;br &amp;gt;&lt;br /&gt;
:* [[Compiling Your Code| Compiling Your Code]]&amp;lt;br &amp;gt;&lt;br /&gt;
:* [[Running Your Code| Running Your Code]]&amp;lt;br &amp;gt;&lt;br /&gt;
:* [[Graphical Applications| Graphical Applications]]&amp;lt;br &amp;gt;&lt;br /&gt;
&lt;br /&gt;
TRACC Cluster Software&lt;br /&gt;
:* [[TRACC Cluster Software#Supported On ARROW| Supported On ARROW]]&amp;lt;br &amp;gt;&lt;br /&gt;
:* [[TRACC Cluster Software#Under Investigation| Under Investigation]]&amp;lt;br &amp;gt;&lt;br /&gt;
&lt;br /&gt;
TRACC Policies&lt;br /&gt;
:* [[TRACC Policies#Application Restrictions| Application Restrictions]]&amp;lt;br &amp;gt;&lt;br /&gt;
:* [[TRACC Policies#Data Retention| Data Retention]]&amp;lt;br &amp;gt;&lt;br /&gt;
:* [[TRACC Policies#Email| Email]]&amp;lt;br &amp;gt;&lt;br /&gt;
:* [[TRACC Policies#Login Nodes| Login Nodes]]&amp;lt;br &amp;gt;&lt;/div&gt;</summary>
		<author><name>Ley</name></author>
	</entry>
</feed>