HPC/Submitting and Managing Jobs/Queues and Policies

Introduction

There is one main queue and one debug queue on Carbon, defined as Torque queues. Once submitted, job routing decisions are made by the Moab scheduler.

In this framework, short jobs are accommodated by a daily reserved node and by backfill scheduling, i.e. "waving forward" small jobs while one or more big jobs wait for full resources to become available.

Default queue "batch"

The main queue on Carbon is batch and need not be specified to qsub or in the job script. The following defaults and limits apply:

resources_default.nodes = 1:ppn=8
resources_default.walltime = 00:15:00    # 15 min
resources_max.walltime = 240:00:00       # 10 days
max_user_queuable = 2000

And, given current node hardware:

ppn ≤ 8

In addition, the Moab scheduler applies various per-user limits. Straightforward are hard and soft limits on the number of concurrent jobs and CPU cores used (about 60% of the whole machine), designed to prevent monopolizing the cluster by a single user while permitting use of otherwise idle resources. A more advanced parameter is a cutoff for queued jobs considered for scheduling, based on their total number of cores requested (MAXIPROC). This ensures a fair job turnover between different users, while not restricting throughput for large numbers of "small" jobs.

See also:

• Torque queue configuration.
• Moab Usage Limits/Throttling Policies.

Queue "debug"

For testing job processing and your job environment, use qsub -q debug on the command line, or the follwing in a job script:

#PBS -q debug

The debug queue accepts jobs under the following conditions

resources_default.nodes = 1:ppn=4
resources_max.nodes = 2:ppn=4
resources_default.walltime = 00:15:00
resources_max.walltime = 01:00:00
max_user_queuable = 3
max_user_run = 2

in other words,

nodes ≤ 4
ppn ≤ 4
walltime ≤ 1:00:00    # 1 hour