HPC/FAQ

From CNM Wiki
< HPC
Jump to navigation Jump to search

Access and proposals

The HPC Carbon cluster and its associated software licenses may be accessed, like any tool at the CNM, by one or both of the following routes:

  • CNM Facility Users are eligible under CNM's User Access Program. Follow the link to learn how to initiate access under this route. Begin by registering as Facility User, then submit a proposal.
  • Affiliates of Argonne's NST division are eligible for discretionary access. Affiliate here refers to regular employees, Argonne/University joint appointees, visitor appointees, and student group members, being supervised by CNM staff.

The following is a summary of the main points as they pertain to Carbon.

Legal User Agreement

The institution where you work or study must have a legal signed User Facility Agreement with Argonne in place before any work under an eventual proposal may be performed.

Look up existing User Facility Agreements and locate your institution if you have never worked with Argonne before, or you recently changed your affiliation, or are going to change it soon.

If your institution is not shown, get started with establishing a user agreement as soon as possible. Legal wheels turn slowly and your scientific work could be delayed.

User Registration

Under CNM's User Access Program, your starting point is to become a CNM user.

Proposal Lifecycle Overview

  1. To submit a proposal, follow the instructions at the CNM's Call for Proposals.
  2. Upon successful review of a proposal and grant of an allocation, the contacting author will receive an email with subject User Proposal Status Notification.
  3. The contacting author must submit, for each proposal, a Safety&Data Form as instructed in the status email. This will confirm the names and badge numbers of participating users, as well as various characteristics of the proposal that we need for statistical reporting to our funding agencies.
  4. Participating users register and respond at various steps to have their Argonne user accounts created, activated or reactivated as needed, and have them authorized to access Carbon.
  5. Participating users perform the work proposed.
  6. Credit CNM in publications that result from your proposal work with us.
  7. After the proposal expires, our User Office will ask you to please submit:
    • An approximately 1-page Activity Report summarizing work performed and perhaps already published under the proposal. We use these reports to document the use of our facilities to funding agencies.
    • An end-of-proposal Satisfaction Survey, to invite your feedback on our process.

Account Types

You access our computer systems by two different user accounts:

User Registration Account

– Looks like: 123456 –

Your User Registration Account:

  • … is created when you register as CNM facility user, be it as participant or as Principal Investigator (PI), at the User Registration for the Advanced Photon Source (APS) and CNM.
  • … has an all-numeric user name (no letters, no leading zeroes), which never changes – this is in fact your Argonne Badge Number,
  • … has its own password and password reset process at APS,

Use your User Registration Account to:

Argonne Domain Account

– Looks like: b123456 or jsmith –

Your Argonne Domain Account:

  •  … is created when you first need it, such as when you are a PI and have received the go-ahead on your first proposal with us,
  • … has, at first, a user name beginning with b, followed by your badge number (without any leading zeroes),
  • … will get, following your request, a personalized user name, such as jsmith
  • … has its own password and uses its own password reset and recovery.

Use your Argonne Domain Account to:

  • … submit the Safety&Data Form (UWS) of each proposal for which you are the PI or delegated spokesperson,
  • … perform work – always remotely – at CNM's High Performance Computing system Carbon,
  • … handle almost all other tasks at Argonne.

For assistance, please contact our CNM User Office. – Specify which task you have difficulties with.

Getting started on a proposal

To start working on a proposal once it has been granted, action is required for each proposal by the proposal spokesperson (usually the Principal Investigator,PI), regardless of any previous proposals with us. At this stage, we collect and update information about each proposal for reporting purposes to our funding agencies, and ensure that safety and training requirements are met by all participating users.

To get started as spokesperson or delegate

  1. Locate the email to you with subject User Proposal Status Notification, sent by the CNM User Office.
    • If you are delayed in starting your proposal, you may need to expand your mailbox search time frame, to begin the search at possibly several months before present.
    • You may need to inspect the "Junk" or "Spam" folder of your mail application.
  2. Follow the instructions in that email.
    • Certain steps are needed only once because they pertain to your institution, notably filing a User Agreement.
    • Before work on a new proposal may begin, we may ask you to submit Activity Reports that are due for previous proposals upon their conclusion.
    • Fill in the Safety and Data form, i.e., follow the User Work Submittal (UWS) link that is given in our notification email.
    • Only the spokesperson may fill in the UWS form, or the PI may designate a delegate. Changes after the initial UWS submission by you (the PI or delegate) can only be made by your Scientific Contact at the CNM – see below.
    • The UWS is due for each proposal, as it collects certain metadata that we need, on a per-proposal-basis, for usage reporting to our funding agencies.
    • Confirm or add users in section "Personnel Participating In User Proposal" of the UWS form. Enter the badge numbers for all participating users, as far as they are known at submission time.
  3. Direct all users who you expect to access Carbon, possibly including yourself, to follow the instructions in section #Ongoing user access requirements.
  4. Be patient. UWS Processing typically takes at least one business day, as both our User Office staff and your Scientific Contact need to attend to your submission in person.
  5. The User Office will notify the spokesperson or delegate once work on a proposal may begin, issuing what we call the User Work Approval (UWA). It will be valid for one year for regular proposals, and 6 months for rapid-access proposals.

To get started as participating user

For new proposals or users new to CNM:

  1. Gently remind your spokesperson/PI to submit the UWS as shown above, or ask this person to nominate you via an email to the CNM User Office.
  2. Continue at section #Actions required by the User.

For proposals already in progress, see the next section.

Adding users to a proposal

Most users are nominated to work on a proposal when it is first submitted. Additional users can be authorized to work under a proposal at any time after acceptance. Actions are needed from both the user and the proposal spokesperson.

Actions required by the User

1. Enter or review your registration as facility user and update as necessary.
2. Follow up in a day or a week as specified at #Follow-up actions required by the User.

Actions required by the PI or spokesperson

3. Confirm that the user has entered or reviewed their registration, and determine their badge number(s).
4. Add the user to the proposal.
  • If the proposal has not yet started: See the section above #To get started as spokesperson or delegate.
  • If the proposal has already started: Email the users' names and badge numbers to the proposal's Scientific Contact at the CNM and ask for the UWS to be augmented.

Actions required by the Scientific Contact

5. Open the UWS-SciCon form in the CNM Proposal Dashboard on the "Inside CNM" web page.
  • Add users under section "Participating Personnel".
  • Enter users' badge numbers which may have been left empty in earlier UWS submissions, as happens by necessity for very newly registered users.
  • Again, review that badge numbers are present and are correct for all relevant participants that are to work on Carbon, or at least need to use mega.

Follow-up actions required by the User

6. Contact our CNM User Office and ask to review your registration and to initiate the next steps.
  • To work onsite at CNM or remotely on Carbon, you will need an Argonne computer account, which is different from the account you used to enter your User Registration details – see section #Account Types.
  • If you are new to Argonne, our User Office will be in contact with you to set up this account. Since the Argonne account is for more general use than your registration account, it will involve formal approvals, cyber security training, etc. and typically takes about one week to complete.
7. Complete or refresh training courses, as requested by the User Office.
  • Course requirements are triggered at various levels. There are courses for being a visitor at Argonne, a user at CNM, and for work performed under proposals, be it onsite or remotely.
8. Have your Argonne computer account finalized and ready.

Carbon account and access

When all prerequisites are met, your Argonne account will be enabled or (if so needed) re-enabled for use on Carbon.

Enabling your account on Carbon does not happen immediately after a password reset or rename, or submitting proposal forms – it is a separate process.
9. Await confirmation from us that your account has been enabled on Carbon and been added to one or more proposals.
10. Review and follow the instructions on HPC/Network Access, i.e., how to connect to the cluster.

If you do not receive an expected notification within two days, contact our User Office and describe your issue.

Finalize your Argonne computer account

After your registration as a new Argonne Facility User has been processed, an Argonne "domain account" will be created for you. Its user name will be in the form b123456 and our User Office will have communicated to you an initial password.

To begin work on Carbon and possibly other CNM instruments, personalize both your password and your user name, in this order.

For Argonne personnel: User name vs. email address
  • Your user name is the one by which you log in to Argonne-internal services, and takes one of three forms. Web login services typically indicate which user name form they expect..
    • username –  This is the most typical form expected, used to log in to mega.
    • [email protected] – Your user name with a directory domain appended.
  • ANL.GOV\username – An alternate form of your user name and directory domain.
  • Your user name is not necessarily merely the name part of an Argonne email address that you will have if you work or study at Argonne.

Change your password

  • Change your password as instructed when your account got created.
    See also section #Verify or change your password below.
  • If you chose to call the Argonne Help Desk at +1-630-252-9999, option 4, to make the password change:
    While on the phone regarding your password, state that you are a CNM Carbon User and ask to also personalize your user name.

Change your user name

  1. Request a user name change, either over the phone or by email. We will choose the user name for you.
    To request by phone
    Call the Argonne Help Desk at +1-630-252-9999, and select option 4.
    To request by email
    Follow this link: Open email template – (Use a web browser that supports "mailto:" links.)
    1. Respond from the same email address as you filled in at the CNM User Registration form.
      In your email application, choose or switch to the mail account that handles the address you registered with at our User Registration. Messages pertaining to your Argonne activities that are sent from an unrelated email account may be disregarded (not taken up) by our help desk staff, for obvious reasons.
    2. Fill in your name and badge number in the email template that opens, then send the email.
  2. Await an emailed ticket number from Argonne's service desk, or follow the instructions of the person you called.
    • Ticket numbers are assigned upon manual review, typically within an hour or so during business hours.
    • If you did not get a ticket number after such reasonable time, re-send the request, making sure you do so from the email account you provided at User Registration.
  3. Check your email for follow-up questions and respond.
  4. Await the completion notice for your user name change.

Next steps

Carbon access is not immediately enabled once your user name change request has been completed – it is a separate process.

  1. Expect an email that your Carbon access has been granted.
  2. Proceed under #Carbon account and access.

Proposal troubleshooting

Email the CNM User Office with any questions or concerns, such as:

  • about proposals,
  • about users, or
  • you find that responding to any of your submissions or previous communications takes longer than a few business days.

Only our User Office staff is able to review all aspects of your proposal or your user access requirements, and determine any steps that have yet to be taken or need to be refreshed.

Include in your message:

  • Proposal number(s),
  • Name of the Principal Investigator (PI),
  • Names and badge numbers of participants.

Ongoing user access requirements

For you to log into Carbon and its SSH gateway, a number of criteria must be met, most of which are subject to expiration dates and require action from you for renewal.

You will not get notified (for various reasons) on some of these expirations. Different duration terms may cause expirations to happen right in the middle of your proposal's lifecycle and can cause immediate inconvenience. Depending on which criterion expired, one or both of the following will happen to your Argonne computer account:

  • The account's permission to use mega will be revoked.
  • The account as such will be disabled, equivalent to revoking all its permissions.

To recover, do one or more of the following:

User Registration

  1. Review and update your User Registration.
    • For this system, log in with your badge number and the password associated with it, not your regular Argonne account – see section #Account Types above.
    • The registration itself requires renewal at least every 2 years, or earlier when any one of various prerequisite items expires.
    • For non-US citizens present in the US: Review and update registration items regarding your US visa or related work permit. A current US work or study status is required to access Argonne computers, the same as if you were to visit in person.
  2. Ask our CNM User Office staff to review your registration updates and to recreate or re-instate your Argonne account, as the case might be.

Legal User Agreement

  • If you changed your affiliation (the institution where you work or study), check to see if your new institution has a legal User Facility Agreement with Argonne in place and request one if not.
    Be advised that the process may take several weeks to percolate through legal and adminstrative channels.

Training

  • Review and renew your User Courses.
    For remote users, the ESH223 course "Cybersecurity Annual Education and Awareness" is the one most likely to be in need of renewal.

Password

  • Review and update the password to your Argonne computer account if you now have or previously had such an account.
    If your account's password expired, we consider the account dormant and will not notify you about new proposals where you are listed as a participant.

Active proposal

You must be a participant in at least one active or recently expired User Proposal. You may run compute jobs under a given proposal while it is active, i.e., within the dates stated in the proposal's User Work Approval (UWA).

  • To review the dates for your proposals, ask your PI to search their email archive for subjects "Work Approval Received" or "Proposal Expiration".

Data-only access for inactive proposals

After your last active proposal on Carbon has expired you may still access, for up to 30 days, the SSH gateway and Carbon's login nodes. Past that time window, your access will be revoked and your data may be deleted, following CNM's Data Retention Policy.

  • While you still have access, offload from Carbon all your files and data that you may wish to keep.
    The CNM cannot be expected or held responsible to store your data beyond your access window.

Practical hints

  • Set yourself calendar entries about one year into the future to remind yourself to renew any of your user registration or training requirements.
  • After you were added to a user proposal, wait at least an hour or more before trying to access our SSH gateway, preferably until the next morning.
    Updates of your status need to be propagated through a handful of systems, each being done about hourly, so it may take several cycles for your status change to reach mega.

Login issues

When you ask "I cannot log in" or "My password does not work", consider the following sections:

Review your access requirements

See section #Ongoing user access requirements above.

Verify or change your password

Do one of the following:

  • Visit https://mypassword.anl.gov/
    • Complete the Password Enrollment process at this service on your first visit when you know your password. You will be (or have been) thus instructed when your account has been created.
    • You can verify or change your existing known password.
    • You can reset or unlock your password only after you enrolled.
    • You cannot use this service if your account has been disabled as such, in which case revisit the section #Ongoing user access requirements.
  • You can change your known password also at our SSH gateway host mega.
    • At least one of your CNM proposals must still be active.
    • You cannot log in if your account is disabled as such.
Error messages for login failures will not disclose a specific reason. You cannot tell if merely your password was wrong or your account is disabled.

After a password update, you should be able to connect to mega normally. If you find that you cannot log in, however, retry only after an hour or two, as in some cases extra time is needed for passwords or other account changes to propagate.

Password lockouts

After trying incorrect passwords several times, your account may be temporarily locked out. To recover, try one of the following:

Review host names

Connect to the correct host names:

  • When connecting from outside Argonne, at least two ssh sessions are required.
    • mega.cnm.anl.gov – an ssh "tunnel setup" connection.
    • carbon – one or more "payload" connections for the ssh, scp, or sftp commands. Here, carbon is not a hostname but an entry in your ssh configuration that you must make (called a host alias or "profile", depending on the ssh application). The entry stands for a connection to localhost (your machine), at a port number forwarded by the preceding tunnel setup connection.
  • When onsite (for any user) or using VPN (for Argonne staff):
    • carbon.cnm.anl.gov. The previous name was clogin. ….

To learn more, read HPC/Network Access.

Review network configuration

Request support

Expected files or commands are unavailable

When you cannot create or edit files or directories under your home directory, or you find that commands you expect are not found, then:

  1. Inspect the beginning of your command line, called the prompt:
    • If your prompt mentions "@mega" or "GATEWAY", e.g.:
    username@mega5 CNM-GATEWAY $
    then you are working not on Carbon just yet, but rather on its SSH gateway. Visit HPC/Network_Access to learn how to connect properly.
    • If the prompt mentions "@login", e.g.:
    username@login6 … $
    then there may be a temporary file system issue which may resolve after a few minutes.
    If such issue persists or recurs over several hours, however, read the message that will have appeared right after you logged in, above the first command prompt. To locate these messages, scroll up the content of your terminal application window, or log out and log back in to bring up these messages again.

My home directory is read-only

I cannot edit my .bashrc file

Expected commands unavailable

In all these situations, see the beginning of the current section.

Mailing lists

Announcements about Carbon are made on the cnm-hpc-announce mailing list, hosted at Argonne. The mailing list home page and its archive are, unfortunately, only accessible from onsite or (for authorized users) over VPN.

  • To unsubscribe from the mailing list, do one of the following:
    1. Open the "unsubscribe" link at the bottom of a recent message that you received from the mailing list.
    2. Send a blank message to cnm-hpc-announce-leave@lists.anl.gov and follow up on the confirmation notice.
  • To subscribe to the mailing list, send a blank message to cnm-hpc-announce-join@lists.anl.gov and follow up on the confirmation notice.
  • To change your email address on the list, simply do both of the above, in order.
Hints
  • When sending commands by email, be sure to have the relevant email account selected in the "From" line of the compose window in your email application.
  • Inspect the Junk mail folder of your email application if you do not receive a confirmation message for subscribing or unsubscribing within a minute or so.
  • See the GNU Mailman documenation for background.

Applications

I'd like to use application X

Check if the application is already available on Carbon

Either: 

module avail
module -l avail 2>&1 | less
The second form gives you browsable output.
If you cannot find the application on Carbon
  • Submit a support request.
    • Provide one or more URLs relevant to software you have in mind – be specific.
    • Describe the problem you are trying to solve – it may well be that we can suggest an alternative solution.
    • Give the extent of your planned use.
If you see the application on Carbon but you cannot access it
  • Existing license agreements may cover only a subset of users (typically Argonne employees).
  • If you feel you are eligible, submit a support request.
If a version newer than the installed one on Carbon is available

How do I run application X?

  • Customize your shell environment to load the application module.
  • Learn about module conventions on Carbon.
  • To determine the names of a package's executable scripts and binaries, load the application module (if you have not yet done so in your shell setup), then inspect the module's $NAME_HOME/bin directory. For instance, for the Quantum-ESPRESSO package:
module load quantum_espresso
ls $QUANTUM_ESPRESSO_HOME/bin

How do I use application X?

Read the package's documentation, using one or more of the following:

  • Inspect the package's $NAME_HOME/share or $NAME_HOME/doc directory on Carbon (see module conventions).
  • Browse the package's web page, generally mentioned in the module help text or the application catalog entry.
  • Consult a package's man pages. Few packages have them. Man page files are generally installed under $NAME_HOME/man or $NAME_HOME/share/man and if so, will be made available automatically to the man command.

What's my account balance?

Simple answer: mybalance

To find out how many core-hours you have available, the simplest command to run is: 

mybalance -h

Project  Machines Balance    
-------- -------- ---------- 
user     ANY         993.26
cnm34567 ANY       158760.93
cnm31234 ANY      -148893.62

The table gives all the Projects you have access to (for use with the qsub -A argument), and their balance. Machine lists all systems that can book jobs against your allocations. Carbon is currently the only machine that can do so. Balance is your account balance, in core-hours, as selected by the -h command option. This is the most useful and recommended unit. Without -h, you get core-seconds, which are integers but rather more unwieldy numbers.

  • The "user" project provides you with a small initial startup allocation of typically 1000 core-hours.
  • When a Balance is reported as negative, that account typically has a CreditLimit assigned, which permits the balance to dip below zero. These details, however, are not shown by mybalance.

Complete answer: gbalance

To get allocation details for accounts that have CreditLimits, run the gbalance command. Pass on -u username or -p projectname to select your allocations: 

gbalance -h -u $USER
Type $USER as shown. The command interpreter will fill in your actual username.

The ouput looks like: 

Id  Name     Amount     Reserved Balance    CreditLimit Available
--- -------- ---------- -------- ---------- ----------- --------- 
100 cnm31234 -148893.62     0.00 -148893.62   150000.00   1106.38
217 kpelzer      993.26     0.00     993.26        0.00    993.26 
123 cnm34567  166440.93  7680.00  158760.93        0.00 158760.93

The most relevant column for you is Available. The units, given the -h option, are again core-hours.

The colums and their meanings are:

Id
an internal number for the account.
Name
The project name (for use with qsub -A or #PBS -A).
Amount
Amount for transactions completely on the books for the project account; does not include running jobs or credits. Deposits are allocated by the User Office and implemented by the Carbon administrator.
Reserved
Amounts held in reserve by all running jobs using this account. The reserve ensures that a job does not cause an overdraft when it finishes and when its actual use will be booked. The quantity is calculated by walltime * number of cores blocked. When a job terminates, the charge according to the actual time used will be subtracted from Amount, and the unused quantities will be re-added to Available.
Balance
Available for new jobs; may go negative if CreditLimits are in place.
Balance = Amount - Reserved
CreditLimit
Amount by which Balance may go negative; assigned by the Carbon administrator.
Available
Relevant quantity for new jobs. Must be positive for a new job to start, and large enough to Reserve the entire job.
Available = Balance + CreditLimit

Allocation expiration policy – Or: Why did my account balance suddenly drop?

The compute time physically available by Carbon's processors is a perishable resource. Hence, your allocations are time-restricted in a use-it-or-lose-it manner. This is done to encourage consistent use of the machine throughout allocation cycles.

Because of resource contention, especially near the end of a cycle, it will be increasingly impractical and eventually physically impossible to use up a large remaining allocation within a short time. You would need to claim a large fraction of Carbon's nodes during a relatively short time window, which is unlikely to be possible because jobs from other users will be running as well.

The expiration schedule is as follows:

  • Your allocation is provided in three equal-sized installments.
  • All installments are active from the beginning.
  • Installments expire in a staggered fashion, currently after 4, 8, and 12 months, respectively. A diagram might illuminate this:
    Proposal                Proposal
     start                 expiration
       |-------|-------|-------|------> Time    
       0       4       8       12      (months)
       |
Installment
       |
    (1)|########....................               KEY
       |                            
    (2)|################............               .  Installment is inactive
       |                            
    (3)|########################....               #  Installment is active
       |
  • Your jobs will, sensibly, be booked against the installments that expire the earliest.

My question is not answered here

See HPC/Support.

Retrieved from "https://wiki.anl.gov/wiki_cnm/index.php?title=HPC/FAQ&oldid=7240"