HPC/Submitting and Managing Jobs - Revision history

Stern: /* Inspecting stuck jobs */

2026-02-17T20:45:29Z

Inspecting stuck jobs

← Older revision		Revision as of 20:45, February 17, 2026
Line 75:		Line 75:

	To get estimates on when your job might start:		To get estimates on when your job might start:
	showstart ~~[-e all]~~ ''jobnum''		showstart ''jobnum''

	== Inspecting running jobs ==		== Inspecting running jobs ==

Stern: /* Inspecting stuck jobs */

2026-02-17T20:44:42Z

Inspecting stuck jobs

← Older revision		Revision as of 20:44, February 17, 2026
Line 70:		Line 70:
	== Inspecting stuck jobs ==		== Inspecting stuck jobs ==
	To diagnose problems with "stuck" jobs, use <code>qstat -s</code>, or simply Moab's		To diagnose problems with "stuck" jobs, use <code>qstat -s</code>, or simply Moab's
	[http://docs.adaptivecomputing.com/9-1-3/suite/help.htm#topics/moabWorkloadManager/moabCommands/commands/checkjob.html~~?Highlight=checkjob~~ checkjob] command. The last few lines of output give ''past events'' of the job and its current status.		[http://docs.adaptivecomputing.com/9-1-3/suite/help.htm#topics/moabWorkloadManager/moabCommands/commands/checkjob.html checkjob] command. The last few lines of output give ''past events'' of the job and its current status.
	qstat -s ''jobnum''		qstat -s ''jobnum''
	checkjob -v ''jobnum''		checkjob -v ''jobnum''

Stern: /* Inspecting stuck jobs */

2026-02-17T20:44:12Z

Inspecting stuck jobs

← Older revision		Revision as of 20:44, February 17, 2026
Line 69:		Line 69:

	== Inspecting stuck jobs ==		== Inspecting stuck jobs ==
	To diagnose problems with "stuck" jobs, use <code>qstat -s</code>, or simply Moab's [http://~~www~~.adaptivecomputing.com/~~resources~~/~~docs~~/~~mwm~~/~~6-1-9~~/~~Content~~/commands/checkjob.html checkjob] command. The last few lines of output give ''past events'' of the job and its current status.		To diagnose problems with "stuck" jobs, use <code>qstat -s</code>, or simply Moab's
			[http://docs.adaptivecomputing.com/9-1-3/suite/help.htm#topics/moabWorkloadManager/moabCommands/commands/checkjob.html?Highlight=checkjob checkjob] command. The last few lines of output give ''past events'' of the job and its current status.
	qstat -s ''jobnum''		qstat -s ''jobnum''
	checkjob -v ''jobnum''		checkjob -v ''jobnum''

Stern: /* Inspecting "stuck" jobs */

2026-02-17T20:40:49Z

Inspecting "stuck" jobs

← Older revision		Revision as of 20:40, February 17, 2026
Line 68:		Line 68:
	qstat -f [''jobnum''] [-1]		qstat -f [''jobnum''] [-1]

	== Inspecting "stuck" jobs ==		== Inspecting stuck jobs ==
	To diagnose problems with "stuck" jobs, use <code>qstat -s</code>, or simply Moab's [http://www.adaptivecomputing.com/resources/docs/mwm/6-1-9/Content/commands/checkjob.html checkjob] command. The last few lines of output give ''past events'' of the job and its current status.		To diagnose problems with "stuck" jobs, use <code>qstat -s</code>, or simply Moab's [http://www.adaptivecomputing.com/resources/docs/mwm/6-1-9/Content/commands/checkjob.html checkjob] command. The last few lines of output give ''past events'' of the job and its current status.
	qstat -s ''jobnum''		qstat -s ''jobnum''

Stern: /* Submitting and managing jobs */

2026-02-17T20:40:19Z

Submitting and managing jobs

← Older revision		Revision as of 20:40, February 17, 2026
Line 32:		Line 32:
	See also: [[HPC/Directories\|directory layout]].		See also: [[HPC/Directories\|directory layout]].

	== Submitting ~~and managing~~ jobs ==		== Submitting jobs ==
	=== Job Script ===		=== Job Script ===
	A job is described to Torque by a ''job script,'' which in most cases is a shell script containing directives for Torque.		A job is described to Torque by a ''job script,'' which in most cases is a shell script containing directives for Torque.
Line 52:		Line 52:
	See [[HPC/Submitting and Managing Jobs/Queues and Policies \| Queues and Policies]].		See [[HPC/Submitting and Managing Jobs/Queues and Policies \| Queues and Policies]].

	=== Checking job status ===		== Checking job status ==
	Query your jobs using commands similar to those on other HPC clusters.		Query your jobs using commands similar to those on other HPC clusters.
	You can use either [http://www.adaptivecomputing.com/resources/docs/torque/4-1-4/Content/topics/12-appendices/commandsOverview.htm Torque commands]		You can use either [http://www.adaptivecomputing.com/resources/docs/torque/4-1-4/Content/topics/12-appendices/commandsOverview.htm Torque commands]
Line 68:		Line 68:
	qstat -f [''jobnum''] [-1]		qstat -f [''jobnum''] [-1]

	==~~= Inspect~~ stuck jobs ===		== Inspecting "stuck" jobs ==
	To diagnose problems with "stuck" jobs, use <code>qstat -s</code>, or simply Moab's [http://www.adaptivecomputing.com/resources/docs/mwm/6-1-9/Content/commands/checkjob.html checkjob] command. The last few lines of output give ''past events'' of the job and its current status.		To diagnose problems with "stuck" jobs, use <code>qstat -s</code>, or simply Moab's [http://www.adaptivecomputing.com/resources/docs/mwm/6-1-9/Content/commands/checkjob.html checkjob] command. The last few lines of output give ''past events'' of the job and its current status.
	qstat -s ''jobnum''		qstat -s ''jobnum''
Line 76:		Line 76:
	showstart [-e all] ''jobnum''		showstart [-e all] ''jobnum''

	==~~= Inspect~~ running jobs ===		== Inspecting running jobs ==
	==== Peeking at job output files ====
			=== Peeking at job output files ===
	Each job has associated stdin and stderr streams which are gathered on its first node and are copied back only at job end time, according to the <code>#PBS</code> directives -e, -o, or -j [eo\|oe]. If the user does explicitly capture any output in the script (using shell redirections <code>">"</code>) it can be difficult to assess a job's progress.		Each job has associated stdin and stderr streams which are gathered on its first node and are copied back only at job end time, according to the <code>#PBS</code> directives -e, -o, or -j [eo\|oe]. If the user does explicitly capture any output in the script (using shell redirections <code>">"</code>) it can be difficult to assess a job's progress.

Line 91:		Line 92:
	qpeek -p ''jobnum''		qpeek -p ''jobnum''

	==== Which nodes does a running job use? ====		=== Which nodes does a running job use? ===
	qstat -n [-1] ''jobnum''		qstat -n [-1] ''jobnum''

	=== Changing ~~jobs~~ after submission ===		== Changing job resources after submission ==
	~~Use~~ <~~code>[~~http://www.adaptivecomputing.com/resources/docs/torque/4-1-4/Content/topics/commands/qalter.htm qalter]</code> to modify attributes of a job.		<!-- http://www.adaptivecomputing.com/resources/docs/torque/4-1-4/Content/topics/commands/qalter.htm -->
			Use <code>qalter</code> to modify attributes of a job.
	Only attributes specified on the command line will be changed.		Only attributes specified on the command line will be changed.
	If any of the specified attributes cannot be modified for any reason, ''none'' of that job's attributes will be modified.		If any of the specified attributes cannot be modified for any reason, ''none'' of that job's attributes will be modified.
	qalter [-l ''resource''] […] ''jobnumber'' …		qalter [-l ''resource''] […] ''jobnumber'' …
	qalter -A ''accountspec'' ''jobnumber'' …		qalter -A ''accountspec'' ''jobnumber'' …
	* Most attributes can be changed only ''before'' the job starts. Others, notably [http://www.adaptivecomputing.com/resources/docs/torque/4-1-4/Content/topics/commands/qsub.htm#-m mail options] and [http://www.adaptivecomputing.com/resources/docs/torque/4-1-4/Content/topics/commands/qsub.htm#-W job dependencies] can be meaningfully adjusted after start.		* Most attributes can be changed only ''before'' the job starts. Others, notably
			<!-- http://www.adaptivecomputing.com/resources/docs/torque/4-1-4/Content/topics/commands/qsub.htm#-m --> mail options and <!-- [http://www.adaptivecomputing.com/resources/docs/torque/4-1-4/Content/topics/commands/qsub.htm#-W --> job dependencies can be meaningfully adjusted after start.

	; Advanced usage:		; Advanced usage:
Line 107:		Line 110:
	Contact me ([[User:Stern\|stern]]) to arrange for interventions.		Contact me ([[User:Stern\|stern]]) to arrange for interventions.

	==~~= Removing~~ jobs ===		== Canceling (removing) jobs ==
	To retract a queued job or kill an already running job:		To retract a queued job or kill an already running job:
	qdel ''jobnumber''		qdel ''jobnumber''
	[http://www.adaptivecomputing.com/resources/docs/torque/4-1-4/Content/topics/12-appendices/prologueAndEpliogueScripts.htm Torque epilogue scripts] will immediately remove remaining user processes and user files from the used compute nodes upon job termination, be it natural or after qdel.		<!-- http://www.adaptivecomputing.com/resources/docs/torque/4-1-4/Content/topics/12-appendices/prologueAndEpliogueScripts.htm -->
			Torque epilogue scripts will immediately remove remaining user processes and user files from the used compute nodes upon job termination, be it natural or after qdel.

	== Declaring that a job requires licenses ==		== Declaring that a job requires licenses ==

Stern: /* Submitting and managing jobs */

2026-02-17T20:35:44Z

Submitting and managing jobs

← Older revision		Revision as of 20:35, February 17, 2026
Line 58:		Line 58:
	Both access the same information, but through different routes, and offer different display styles.		Both access the same information, but through different routes, and offer different display styles.

	To get the regular output ~~format~~:		* To get the regular status output:
	[http://www.adaptivecomputing.com/resources/docs/torque/4-1-4/Content/topics/commands/qstat.htm qstat] [-u $USER]		[http://www.adaptivecomputing.com/resources/docs/torque/4-1-4/Content/topics/commands/qstat.htm qstat] [-u $USER]
	[http://www.adaptivecomputing.com/resources/docs/mwm/6-1-9/Content/commands/showq.html showq] [-u $USER]		[http://www.adaptivecomputing.com/resources/docs/mwm/6-1-9/Content/commands/showq.html showq] [-u $USER]
	To get an ''alternate'' format (e.g. showing job names):		* To get an ''alternate'' format (e.g. showing job names):
	qstat -a [-u $USER]		qstat -a [-u $USER]
	showq -n [-u $USER]		showq -n [-u $USER]
	~~You can always use~~ <code>-u $USER</code> ~~to show~~ only your jobs.		: Option <code>-u $USER</code> shows only your jobs. You can literally type <code>$USER</code> instead of your user name.
			* To get ''full'' job status information such as submit arguments and run directories, and do not wrap long lines ("-1" is the digit "one"):
			qstat -f [''jobnum''] [-1]

	===~~= Getting extra information =~~===		=== Inspect stuck jobs ===
	~~Show the nodes where a job runs:~~		To diagnose problems with "stuck" jobs, use <code>qstat -s</code>, or simply Moab's [http://www.adaptivecomputing.com/resources/docs/mwm/6-1-9/Content/commands/checkjob.html checkjob] command. The last few lines of output give ''past events'' of the job and its current status.
	~~qstat -n [-1] ''jobnum''~~
	~~Give ''full'' information such as submit arguments and run directories, and do not wrap long lines ("-1" is the digit "one"):~~
	~~qstat -f [''jobnum''] [-1]~~
	To diagnose problems with "stuck" jobs, use ~~the~~ <code>-s</code> ~~option~~, or simply Moab's [http://www.adaptivecomputing.com/resources/docs/mwm/6-1-9/Content/commands/checkjob.html checkjob] command. The last few lines of output give ''past events'' of the job and its current status.
	qstat -s ''jobnum''		qstat -s ''jobnum''
	checkjob -v ''jobnum''		checkjob -v ''jobnum''

	To get estimates on when your job might start:		To get estimates on when your job might start:
	showstart [-e all] ''jobnum''		showstart [-e all] ''jobnum''

			=== Inspect running jobs ===
	==== Peeking at job output files ====		==== Peeking at job output files ====
	Each job has associated stdin and stderr streams which are gathered on its first node and are copied back only at job end time, according to the <code>#PBS</code> directives -e, -o, or -j [eo\|oe]. If the user does explicitly capture any output in the script (using shell redirections <code>">"</code>) it can be difficult to assess a job's progress.		Each job has associated stdin and stderr streams which are gathered on its first node and are copied back only at job end time, according to the <code>#PBS</code> directives -e, -o, or -j [eo\|oe]. If the user does explicitly capture any output in the script (using shell redirections <code>">"</code>) it can be difficult to assess a job's progress.
Line 90:		Line 90:
	To inspect a job's processes running on its compute nodes:		To inspect a job's processes running on its compute nodes:
	qpeek -p ''jobnum''		qpeek -p ''jobnum''

			==== Which nodes does a running job use? ====
			qstat -n [-1] ''jobnum''

	=== Changing jobs after submission ===		=== Changing jobs after submission ===

Stern: /* Declaring that a job requires licenses */

2023-05-12T21:10:57Z

Declaring that a job requires licenses

← Older revision		Revision as of 21:10, May 12, 2023
Line 114:		Line 114:

	'''Note:''' License availability checks do not happen instantaneously but rather by polling, roughly once every minute.		'''Note:''' License availability checks do not happen instantaneously but rather by polling, roughly once every minute.
	: This means that if you or another user happen to start a license-consuming application after the last check but before a newly started job tries to access the same license, that job will terminate with a license failure.		: This means that if you or another user happen to start a license-consuming application after the last check but before the relevant newly started job tries to access the same license, that job will terminate with a license failure.

	=== One license type ===		=== One license type ===

Stern: /* Declaring that a job requires licenses */

2023-05-12T21:09:13Z

Declaring that a job requires licenses

← Older revision		Revision as of 21:09, May 12, 2023
Line 114:		Line 114:

	'''Note:''' License availability checks do not happen instantaneously but rather by polling, roughly once every minute.		'''Note:''' License availability checks do not happen instantaneously but rather by polling, roughly once every minute.
	: This means that if you or another user happen to start a license-consuming application ~~between~~ the ~~time~~ the ~~scheduler sees a~~ license ~~as being available and the time that a job that needs it as well has actually started~~, that job will ~~quickly fail for lack of~~ a license.		: This means that if you or another user happen to start a license-consuming application after the last check but before a newly started job tries to access the same license, that job will terminate with a license failure.

	=== One license type ===		=== One license type ===

Stern: /* Declaring that a job requires licenses */

2023-05-12T21:06:59Z

Declaring that a job requires licenses

← Older revision		Revision as of 21:06, May 12, 2023
Line 114:		Line 114:

	'''Note:''' License availability checks do not happen instantaneously but rather by polling, roughly once every minute.		'''Note:''' License availability checks do not happen instantaneously but rather by polling, roughly once every minute.
	: This means that if you or another user happen to start a license-consuming application between the time the scheduler sees a license as available and the time ~~the~~ job ~~needing~~ it actually ~~starts~~, that job will quickly fail for lack of a license.		: This means that if you or another user happen to start a license-consuming application between the time the scheduler sees a license as being available and the time that a job that needs it as well has actually started, that job will quickly fail for lack of a license.

	=== One license type ===		=== One license type ===

Stern: /* Setting a job to depend on licenses */

2023-05-12T21:04:21Z

Setting a job to depend on licenses

← Older revision		Revision as of 21:04, May 12, 2023
Line 109:		Line 109:
	[http://www.adaptivecomputing.com/resources/docs/torque/4-1-4/Content/topics/12-appendices/prologueAndEpliogueScripts.htm Torque epilogue scripts] will immediately remove remaining user processes and user files from the used compute nodes upon job termination, be it natural or after qdel.		[http://www.adaptivecomputing.com/resources/docs/torque/4-1-4/Content/topics/12-appendices/prologueAndEpliogueScripts.htm Torque epilogue scripts] will immediately remove remaining user processes and user files from the used compute nodes upon job termination, be it natural or after qdel.

	== ~~Setting~~ a job ~~to depend on~~ licenses ==		== Declaring that a job requires licenses ==
	A job that is to run software that requires a commercial license must declare such to the job scheduler (software that decides when and on which nodes the job could start).		A job that is to run software that requires a commercial license must declare such to the job scheduler (software that decides when and on which nodes the job could start).
	The job scheduler monitors known licenses for availability, and so will know which and how many licenses are available at any given time.		The job scheduler monitors known licenses for availability, and so will know which and how many licenses are available at any given time.