HPC/FAQ

Access and proposals

The HPC Carbon cluster may be accessed, like any tool at the CNM, by one of two routes:

• Facility Users, which includes all of Argonne staff, are eligible for CNM's User Access Program.
• As a secondary route, employees of Argonne's NST division are eligible for discretionary access.

Your hosting institution must have a legal signed User Facility Agreement with Argonne in place before any work under an eventual proposal may be performed. Look up your institution under existing Agreements, and get one started right away if not shown — legal wheels turn slowly and can delay your work.

Under the User Program, your starting point is to become a CNM user. To submit a new proposal, follow the instructions at the CNM's Call for Proposals. After a proposal has been successfully reviewed and been granted an allocation, the contacting author will get notified and must then review and complete certain details such as the names of participating users. At the end of the process you or your users will be able to log in using their existing or newly created individual Argonne computer user account.

The following is a summary of the main points as they pertain to Carbon.

Getting started on a proposal

To start working on a proposal once it has been granted, action is required for each proposal by the proposal spokesperson (usually the Principal Investigator,PI), regardless of any previous proposals with us. At this stage, we collect and update information about each proposal for reporting purposes to our funding agencies, and ensure that safety and training requirements are met by all participating users.

To get started as spokesperson or delegate

1. Locate the email that we sent to you with subject User Proposal Status Notification, and follow the instructions therein. Some steps are needed only once because they pertain to your institution.
   • Before work on a new proposal may begin, we may ask you to submit User Activity Highlights that are due for previous proposals upon their conclusion.
   • Your key action, due for each new proposal, will be to fill in the Safety and Data form – follow the User Work Submittal (UWS) link given in the email.
2. Direct all users who you expect to access Carbon, possibly including yourself, to follow the instructions in section #Ongoing user access requirements.
3. Please be patient. Processing typically takes at least one business day, as both our User Office staff and your Scientific Contact need to attend to your submission in person.

The User Office will notify the spokesperson or delegate once work on a proposal may begin. This is what we call the User Work Authorization (UWA), which is typically valid for one year for regular proposals.

To get started as participating user

For new proposals or users new to CNM:

1. Gently remind your spokesperson/PI to submit the UWS as shown above, or ask this person to nominate you via an email to the CNM User Office.
2. Enter, review, or update your User Registration. For returning visits, use your badge number to log in.
3. To work on Carbon, you will need an Argonne computer account, which is different from the user registration account of the preceding item.

   If you are new to Argonne, our User Office will be in contact with you to set up this account. Since the account is for more general use than your registration, it will involve formal approvals, cyber security training, etc. and typically takes about one week to complete. The account will be considered ready after you chose your own password for it.

For proposals already in progress, see the next section.

Adding users to a proposal

Most users are nominated to work on a proposal when it is first submitted. Additional users can be authorized to work under a proposal at any time after acceptance. Actions are needed from both the user and the proposal spokesperson.

Actions required by the User

1. Enter or review your registration as facility user and update as necessary.

2. Follow up in a day or a week as specified below.

Actions required by the PI or spokesperson

3. Confirm that the user has entered or reviewed their registration, and determine their badge number(s).

4. Have the user added to the proposal. How to do this depends on whether the proposal has already started or not (see above).

• For proposals that have already started, submit the users' names and badge numbers to the proposal's Scientific Contact at the CNM.
• If the proposal hasn't started yet, add the users yourself when you initially fill in the User Work Submittel (UWS) as part of the regular proposal startup process. The UWS can and must be submitted by the spokesperson, once only, to start a proposal. To make changes after your submission, ask your Scientific Contact at the CNM.

Your Scientific Contact and the link to the UWS for each proposal are shown in our email to you with the subject containing User Proposal Status Notification.

Actions required by the Scientific Contact

5. Add users in the section "Participating Personnel" in the UWS-SciCon form. Ensure badge numbers are present and correct for users to be working on Carbon and on mega.

• Some users' badge numbers can and may have been left empty in earlier UWS submissions, which happens by necessity for very newly registered users.

Follow-up actions required by the User

6. Contact our CNM User Office and ask to review your registration and initiate the next steps.

7. Complete required training courses as instructed by the User Office.

8. Have your Argonne computer account name and password ready.

New users

  You will need to personalize both your password and user name, in this order.

• When all your registration requirements are met the User Office will assign and send you an initial password. Change the password as instructed.
• Your initial user name will be of the form b123456. To personalize this name, follow the instructions that you will receive by email (usually later in the day) after you chose your password.
• Reply to the email, choosing user names as instructed.
• Await an emailed confirmation with a ticket number from the Argonne Service Desk. The ticket number is assigned upon manual review, typically within an hour or so during business hours.

Existing users

Check for and if necessary renew your expired password, see below.

9. Await confirmation from us that your account has been enabled or re-enabled on Carbon (if so needed) and has been added to one or more proposals.

If you do not receive a notification within two days, contact our User Office and describe your issue.

Proposal troubleshooting

Email the CNM User Office with any questions or concerns, such as:

• about proposals,
• about users, or
• you find that responding to any of your submissions or previous communications takes longer than a few business days.

Only our User Office staff is able to review all aspects of your proposal or your user access requirements, and determine any steps that have yet to be taken or need to be refreshed.

Include in your message:

• Proposal number(s),
• Name of the Principal Investigator (PI),
• Names and badge numbers of participants.

Ongoing user access requirements

For you to log into Carbon and its SSH gateway, a number of criteria must be met, most of which are subject to expiration dates and require action from you for renewal.

Upon expiration of any of the access criteria, your SSH access will be revoked or your Argonne computer account as such will be disabled. Be aware that for some of the expirations, no notice will be sent to you (for varying reasons). Due to different duration terms, expirations are bound to fall right in the middle of your proposal's lifecycle and can cause immediate inconvenience.

User Registration

• Review and update your User Registration.

  The registration itself requires renewal at least every 2 years. It also expires when any of its prerequisite items expires.

  If you are not a US citizen, note especially the entries regarding your US visa or related work permit. Current status is required to access Argonne computers, just as if you were to visit in person.

• After you updated your user registration, contact the CNM User Office to have your Argonne account re-instated.

Legal User Agreement

• If you changed your affiliation, check to see if your new institution has a legal User Facility Agreement with Argonne in place and request one if not.

  Be advised that the process may take several weeks to percolate through legal and adminstrative channels.

Training

• Review and renew your User Courses.

  For remote users, the ESH223 course "Cybersecurity Annual Education and Awareness" is the one most likely to need renewal.

Password

• Review and update the password to Argonne computer account if you now have or previously had such an account.

  If your account's password expired, we consider the account dormant and will not notify you about new proposals where you are listed as a participant.

Active proposal

You must be a participant in at least one active or recently expired User Proposal. You may run compute jobs under a given proposal while it is active, i.e., within the dates stated in the proposal's User Work Authorization (UWA).

• To review the dates for your proposals, ask your PI to search their email archive for subjects "Work Approval Received" or "Proposal Expiration".

Data-only access for inactive proposals

After your last active proposal on Carbon has expired you may still access, for up to 30 days, the SSH gateway and Carbon's login nodes. Past that time window, your access will be revoked and your data may be deleted, following CNM's Data Retention Policy.

• While you still have access, offload from Carbon all your files and data that you may wish to keep.

  The CNM cannot be expected or held responsible to store your data beyond your access window.

Practical hints

• Set yourself calendar entries about one year into the future to remind yourself to renew any of your user registration or training requirements.
• After you were added to a user proposal, wait at least an hour or more before trying to access our SSH gateway, preferably until the next morning.

  Updates of your status need to be propagated through a handful of systems, each being done about hourly, so it may take several cycles, for your status change to reach mega.

Login issues

When you ask "I cannot log in" or "My password does not work", consider the following sections:

Review your access requirements

See section #Ongoing user access requirements above.

Verify or change your password

1. To verify or change your password, do one of the following:
   • Visit https://mypassword.anl.gov/ to change a known or reset an unknown password. For a reset to succeed, you must have previously completed your Argonne Account Profile. You may not succeed if your account is disabled as such, in which case revisit the section #Ongoing user access requirements.
   • Visit https://credentials.anl.gov/ to verify that your username and password are correct.
   • Log in to our SSH gateway host mega if you have previously used it, and at least one of your proposals is still active. If your user account is still active as such and merely has an expired password, you can update your password at that occasion.
   • If you cannot use our automated systems, request a password reset by email to the CNM User Office, at [email protected].
2. Try your new password at mega. If you find that you cannot log in yet, retry after an hour or two – it can take a while for passwords or other account changes to fully progagate to various systems.
3. Connect to mega with the new password. The system may ask you to choose a new password at this point. Choose wisely and remember it.

Password lockouts

After trying incorrect passwords several times, your account may be temporarily locked out. To recover, try one of the following:

• Wait approximately 30 min.
• Unlock at https://mypassword.anl.gov/ if you have set up your Account Profile there.

Review host names

Connect to the correct host names:

• When connecting from outside Argonne, at least two ssh sessions are required.
  • mega.cnm.anl.gov – an ssh "tunnel setup" connection.
  • carbon – one or more "payload" connections for the ssh, scp, or sftp commands. Here, carbon is not a hostname but an entry in your ssh configuration that you must make (called a host alias or "profile", depending on the ssh application). The entry stands for a connection to localhost (your machine), at a port number forwarded by the preceding tunnel setup connection.
• When onsite (for any user) or using VPN (for Argonne staff):
  • carbon.cnm.anl.gov. The previous name was clogin. ….

To learn more, read HPC/Network Access.

Review network configuration

• Read again HPC/Network Access, and follow the instructions for your platform.

Mailing lists

Announcements about Carbon are made on the cnm-hpc-announce mailing list, hosted at Argonne. These list pages and the archive are, unfortunately, only accessible from onsite or via VPN.

• To update your email address on the list, simply unsubscribe as shown in the next item, then re-subscribe by sending a blank message to cnm-hpc-announce-join@lists.anl.gov.
• To unsubscribe from the mailing list, send a blank message to cnm-hpc-announce-leave@lists.anl.gov and follow up on the confirmation notice.

  See the GNU Mailman documenation for background.

Applications

I'd like to use application X

Check if the application is already available on Carbon

Either:

• Browse the Application Catalog, or
• View the catalog on the Carbon command line:

module avail
module -l avail 2>&1 | less

The second form gives you browsable output.

If you cannot find the application on Carbon

• Submit a support request.
  • Provide one or more URLs relevant to software you have in mind – be specific.
  • Describe the problem you are trying to solve – it may well be that we can suggest an alternative solution.
  • Give the extent of your planned use.

If you see the application on Carbon but you cannot access it

• Existing license agreements may cover only a subset of users (typically Argonne employees).
• If you feel you are eligible, submit a support request.

If a version newer than the installed one on Carbon is available

• Submit a support request.
• Include a URL to information about the new version.

How do I run application X?

• Customize your shell environment to load the application module.
• Learn about module conventions on Carbon.
• To determine the names of a package's executable scripts and binaries, load the application module (if you have not yet done so in your shell setup), then inspect the module's $NAME_HOME/bin directory. For instance, for the Quantum-ESPRESSO package:

module load quantum_espresso
ls $QUANTUM_ESPRESSO_HOME/bin

• Learn how to submit and manage jobs.

How do I use application X?

Read the package's documentation, using one or more of the following:

• Inspect the package's $NAME_HOME/share or $NAME_HOME/doc directory on Carbon (see module conventions).
• Browse the package's web page, generally mentioned in the module help text or the application catalog entry.
• Consult a package's man pages. Few packages have them. Man page files are generally installed under $NAME_HOME/man or $NAME_HOME/share/man and if so, will be made available automatically to the man command.

What's my account balance?

Simple answer: mybalance

To find out how many core-hours you have available, the simplest command to run is:

mybalance -h

Project  Machines Balance    
-------- -------- ---------- 
user     ANY         993.26
cnm34567 ANY       158760.93
cnm31234 ANY      -148893.62

The table gives all the Projects you have access to (for use with the qsub -A argument), and their balance. Machine lists all systems that can book jobs against your allocations. Carbon is currently the only machine that can do so. Balance is your account balance, in core-hours, as selected by the -h command option. This is the most useful and recommended unit. Without -h, you get core-seconds, which are integers but rather more unwieldy numbers.

• The "user" project provides you with a small initial startup allocation of typically 1000 core-hours.
• When a Balance is reported as negative, that account typically has a CreditLimit assigned, which permits the balance to dip below zero. These details, however, are not shown by mybalance.

Complete answer: gbalance

To get allocation details for accounts that have CreditLimits, run the gbalance command. Pass on -u username or -p projectname to select your allocations:

gbalance -h -u $USER

Use the literal string $USER which makes the shell fill in your actual username.

The ouput looks like:

Id  Name     Amount     Reserved Balance    CreditLimit Available
--- -------- ---------- -------- ---------- ----------- --------- 
100 cnm31234 -148893.62     0.00 -148893.62   150000.00   1106.38
217 kpelzer      993.26     0.00     993.26        0.00    993.26 
123 cnm34567  166440.93  7680.00  158760.93        0.00 158760.93 

The most relevant column for you is Available. The units, given the -h option, are again core-hours.

The colums and their meanings are:

Id

an internal number for the account.

Name

The project name (for use with qsub -A or #PBS -A).

Amount

Amount for transactions completely on the books for the project account; does not include running jobs or credits. Deposits are allocated by the User Office and implemented by the Carbon administrator.

Reserved

Amounts held in reserve by all running jobs using this account. The reserve ensures that a job does not cause an overdraft when it finishes and when its actual use will be booked. The quantity is calculated by walltime * number of cores blocked. When a job terminates, the charge according to the actual time used will be subtracted from Amount, and the unused quantities will be re-added to Available.

Balance

Available for new jobs; may go negative if CreditLimits are in place.

Balance = Amount - Reserved

CreditLimit

Amount by which Balance may go negative; assigned by the Carbon administrator.

Available

Relevant quantity for new jobs. Must be positive for a new job to start, and large enough to Reserve the entire job.

Available = Balance + CreditLimit

Allocation expiration policy – Or: Why did my account balance suddenly drop?

The compute time physically available by Carbon's processors is a perishable resource. Hence, your allocations are time-restricted in a use-it-or-lose-it manner. This is done to encourage consistent use of the machine throughout allocation cycles.

Because of resource contention, especially near the end of a cycle, it will be increasingly impractical and eventually physically impossible to use up a large remaining allocation within a short time. You would need to claim a large fraction of Carbon's nodes during a relatively short time window, which is unlikely to be possible because jobs from other users will be running as well.

The expiration schedule is as follows:

• Your allocation is provided in three equal-sized installments.
• All installments are active from the beginning.
• Installments expire in a staggered fashion, currently after 4, 8, and 12 months, respectively. A diagram might illuminate this:

    Proposal                Proposal
     start                 expiration
       |-------|-------|-------|------> Time    
       0       4       8       12      (months)
       |
Installment
       |
    (1)|########....................               KEY
       |                            
    (2)|################............               .  Installment is inactive
       |                            
    (3)|########################....               #  Installment is active
       |

• Your jobs will, sensibly, be booked against the installments that expire the earliest.

My question is not answered here

See HPC/Support.