CNM Wiki - User contributions [en]

HPC/Applications/vasp

2026-07-28T20:06:28Z

Stern: /* Requesting Access */

== Requesting Access ==
VASP versions 4, 5, and 6 are is available on Carbon, subject to confirmation for your group having a license for VASP major versions 4, 5, or 6.

Each VASP user needs to be registered, at VASP.at by name and email address, under the VASP license of a research group, or on a commercial license through MDI.

To use VASP binaries on Carbon, send me (M. Sternberg), the following for you, and possibly other users of your VASP license group:

* Full name
* Email address that the person registered with at vasp.at or MDI.



== Module considerations ==
* Review the available modules for each major version:
** [[HPC/Software/Modules#vasp | vasp]], for VASP 4.x
** [[HPC/Software/Modules#vasp5 | vasp5]], for VASP 5.x
** [[HPC/Software/Modules#vasp5 | vasp6]], for VASP 6.x
* Choose and [[HPC/Software/Module Setup#Shell Customizations|add '''one''' of the following lines to your ~/.bashrc]] file:
module load --auto vasp # sets '''$VASP_HOME'''

module load --auto vasp5 # sets '''$VASP5_HOME'''
module load --auto vasp6 # sets '''$VASP6_HOME'''
: Each module points to an installation directory using an [[HPC/Software/Module Setup#Package home directories|environment variable corresponding to the module name]]. Do not hardcode the directory anywhere, as that location is determined by the module and may change in subsequent versions. There should not be a reason to do so, since <code>$PATH</code> will be managed appropriately.

* Each module provides several binaries (see [[#Binaries|next section]]).
* You do not need to specify a path for vasp for the mpirun statement in the PBS job script.
* The modules are mutually exclusive.
* Potentials are available in <code>/opt/soft/vasp-pot</code>.

To access Graeme Henkelman's script suite, load the [[HPC/Software/Modules#vasp-vtstscripts|vasp-vtstscripts module]]. It works with both the vasp and vasp5 modules above. In either case, choose one of the VTST-patched binaries from the table below. The unpatched vasp binaries will not interoperate with the script suite.

module load vasp-vtstscripts

== Binaries ==
Some options for VASP cannot be configured at runtime (by input data) but must be chosen when compiling.
Since no single set of compile-time choices is appropriate for all uses, a number of binaries are provided, distinguished by their name.

The modules from versions vasp/4.6.35-mkl-13 and vasp5/5.2.12a-mkl-4 onwards, respectively,
contain several binaries built using different sets of preprocessor flags in the Makefile.
Earlier modules only contain two binaries, <code>vasp</code> and <code>vasp-gamma</code>.
Additional binaries are provided for useful preprocessor flag combinations
after applying patches from the group of Graeme Henkelman (UT Austin) to the source code:
* [http://theory.cm.utexas.edu/vtsttools/ Transition state tools for VASP], and
* [http://theory.cm.utexas.edu/vtsttools/bader/ DOS Projection in Bader Volumes].

To verify the binaries available for a module, run the appropriate command, once the module has been loaded:
ls $VASP_HOME/bin # for VASP-4 (previous major version)
ls $VASP5_HOME/bin

Choose a binary appropriate for your task and pass its name to <code>mpirun</code> in the job script.
There is no need for full path names.
Copy a sample job script from:
$VASP_HOME/vasp.job # for VASP-4 (previous major version)
$VASP5_HOME/vasp.job

=== Available binaries and their compile-time flags used ===

This table lists the names of the binaries compiled from
* combinations of preprocessor flags, and
* variants of the source code, primarily enhancing capabilities of ionic updates.

{| class="wikitable" cellpadding="5" style="text-align:left; font-family: monospace;"

|- style="background:#e4e4e4; text-align:center; font-weight: bold; font-family: sans-serif;"
| rowspan="2" | Versions and capabilities
| rowspan="2" | Flags used
| colspan="4" align="center" | Binary names
|- style="background:#e4e4e4; text-align:center; vertical-align:top; font-weight: bold; font-family: sans-serif;"
| Official
| Constrained Cell relaxation1
| Patched for VTST2
| Patched for Bader-projected DOS3
|-
! colspan="6" style="background:#e4e4e4; text-align:center; font-family: sans-serif;" |
|-

! rowspan="4" style="background:#e4e4e4; text-align:center; font-family: sans-serif;" | Conventional, Versions 4.x–5.3
| style="background:#e4e4e4;" | (none)
| style="background:none;" | '''vasp'''
| style="background:#eed;" | '''vasp-cellz'''
| style="background:#ded;" | '''vasp-vtst'''
| style="background:#fee;" | '''vasp-baderdos'''
|-
| style="background:#e4e4e4;" | -DNGZhalf
| style="background:none;" | '''vasp-chargehalf'''
| style="background:#eed;" | '''vasp-cellz-chargehalf'''
| style="background:#ded;" | '''vasp-vtst-chargehalf'''
| style="background:#fee;" | '''vasp-baderdos-chargehalf'''
|-
| style="background:#e4e4e4;" | -DwNGZhalf
| style="background:none;" | '''vasp-gamma'''
| style="background:#eed;" | '''vasp-cellz-gamma'''
| style="background:#ded;" | '''vasp-vtst-gamma'''
| style="background:#fee;" | '''vasp-baderdos-gamma'''
|-
| style="background:#e4e4e4;" | -DwNGZhalf -DNGZhalf
| style="background:none;" | '''vasp-gamma-chargehalf'''
| style="background:#eed;" | '''vasp-cellz-gamma-chargehalf'''
| style="background:#ded;" | '''vasp-vtst-gamma-chargehalf'''
| style="background:#fee;" | '''vasp-baderdos-gamma-chargehalf'''
|-
! rowspan="4" style="background:#e4e4e4; text-align:center; font-family: sans-serif;" | Advanced MD techniques,4 Versions 5.2 and 5.3
| style="background:#e4e4e4;" | -Dtbdyn
| style="background:#ddf;" | '''vasp-tbdyn'''
| rowspan="4" style="background:#f0f0f0; text-align:center; font-style:oblique; font-family: sans-serif;" | –
| rowspan="4" style="background:#f0f0f0; text-align:center; font-style:oblique; font-family: sans-serif;" | –
| rowspan="4" style="background:#f0f0f0; text-align:center; font-style:oblique; font-family: sans-serif;" | not useful
|-
| style="background:#e4e4e4;" | -Dtbdyn -DNGZhalf
| style="background:#ddf;" | '''vasp-tbdyn-chargehalf'''
|-
| style="background:#e4e4e4;" | -Dtbdyn -DwNGZhalf
| style="background:#ddf;" | '''vasp-tbdyn-gamma'''
|-
| style="background:#e4e4e4;" | -Dtbdyn -DwNGZhalf -DNGZhalf
| style="background:#ddf;" | '''vasp-tbdyn-gamma-chargehalf'''

|-
! colspan="6" style="background:#e4e4e4; text-align:center; font-family: sans-serif;" |
|-
! rowspan="5" style="background:#e4e4e4; text-align:center; font-family: sans-serif;" | Version 5.44,5
| style="background:#e4e4e4;" | -Dtbdyn
| style="background:#f0f0f0;" | '''vasp_ncl'''
| rowspan="5" style="background:#eed; text-align:center; font-family: sans-serif; " | (–o–)5
| style="background:#ded;" | '''vasp_vtst_ncl'''
| rowspan="5" style="background:#f0f0f0; text-align:center; font-style:oblique; font-family: sans-serif;" | TBD
|-
| style="background:#e4e4e4;" | -Dtbdyn -DNGZhalf
| style="background:none;" | '''vasp_std'''
| style="background:#ded;" | '''vasp_vtst_std'''
|-
| style="background:#e4e4e4;" | -Dtbdyn -DwNGZhalf -DNGZhalf
| style="background:none;" | '''vasp_gam'''
| style="background:#ded;" | '''vasp_vtst_gam'''
|-
| style="background:#e4e4e4;" | (GPU6)
| style="background:none;" | '''vasp_gpu_ncl'''
| style="background:#ded;" | '''vasp_vtst_gpu_ncl'''
|-
| style="background:#e4e4e4;" | (GPU6) -DNGZhalf
| style="background:none;" | '''vasp_gpu'''
| style="background:#ded;" | '''vasp_vtst_gpu'''

|-
! colspan="6" style="background:#e4e4e4; text-align:center; font-family: sans-serif;" |
|-
! rowspan="4" style="background:#e4e4e4; text-align:center; font-family: sans-serif;" | Version 6 (prelim.)
| style="background:#e4e4e4;" |
| style="background:#f0f0f0;" | '''vasp_ncl'''
| rowspan="3" style="background:#eed; text-align:center; font-family: sans-serif; " | (–o–)5
| style="background:#ded;" | '''vasp_vtst_ncl'''
| rowspan="4" style="background:#f0f0f0; text-align:center; font-style:oblique; font-family: sans-serif;" | TBD
|-
| style="background:#e4e4e4;" |
| style="background:none;" | '''vasp_std'''
| style="background:#ded;" | '''vasp_vtst_std'''
|-
| style="background:#e4e4e4;" |
| style="background:none;" | '''vasp_gam'''
| style="background:#ded;" | '''vasp_vtst_gam'''
|}

1) Stress tensor selectively zeroed in subroutine <code>CONSTR_CELL_RELAX</code> 
2) http://theory.cm.utexas.edu/vtsttools/ . The dimer method was broken for vasp5/5.3.3 prior to 5.3.3p3-mkl-2. 
3) http://theory.cm.utexas.edu/vtsttools/dos/ (now defunct) 
4) http://cms.mpi.univie.ac.at/vasp/vasp/Advanced_MD_techniques.html 
5) [[#Constrained Cell Relaxation|Axis-selective constrained cell relaxation]], controlled via an optional input file <code>CELL_RELAX</code>, is compiled into ''all'' binaries. 
6) http://cms.mpi.univie.ac.at/wiki/index.php/GPU_port_of_VASP 

=== Explanation of flags ===
* File <code>vasp.5.2/makefile.linux_pg</code> in the souce code explains:
<pre>
#-----------------------------------------------------------------------
# possible options for CPP:
# NGXhalf charge density reduced in X direction
# wNGXhalf gamma point only reduced in X direction
# avoidalloc avoid ALLOCATE if possible
# IFC work around some IFC bugs
# CACHE_SIZE 1000 for PII,PIII, 5000 for Athlon, 8000 P4
# RPROMU_DGEMV use DGEMV instead of DGEMM in RPRO (usually faster)
# RACCMU_DGEMV use DGEMV instead of DGEMM in RACC (faster on P4)
# **** definitely use -DRACCMU_DGEMV if you use the mkl library
#-----------------------------------------------------------------------
…
#-----------------------------------------------------------------------
# additional options for CPP in parallel version (see also above):
# NGZhalf charge density reduced in Z direction
# wNGZhalf gamma point only reduced in Z direction
# scaLAPACK use scaLAPACK (usually slower on 100 Mbit Net)
#-----------------------------------------------------------------------
</pre>
* In the new build system (since version 5.4) the direct use of the <code>-D*NG*half</code> options has been replaced by predefined make(1) arguments <code>VERSION={std,gam,ncl}</code>.

=== Common flags ===
-DIFC -DRPROMU_DGEMV -DRACCMU_DGEMV -DCACHE_SIZE=16000 -DMPI -DMPI_BLOCK=8000

== General patches ==

The following modifications have been applied and are active in ''all binaries'':

=== vdW-DF functional ===
The [http://cms.mpi.univie.ac.at/vasp/vasp/vdW_DF_functional_Langreth_Lundqvist_et_al.html vdW-DF functional of Langreth and Lundqvist et al.]
is implemented in all the <code>*-chargehalf</code> binaries of the vasp5 module, and all versions above 5.4.

No specific preparations are needed other than setting the INCAR flags as documented. The binaries will find the required precalculated kernel file <code>vdw_kernel.bindat</code> by means of the environment variable <code>$VASP5_HOME</code> which is set by the module.

This is a Carbon-specific extension.

=== Solvation Model ===
All binaries above build 5.4.1.3-8-* contain the [http://vaspsol.mse.ufl.edu VASPsol] solvation model of the Hennig group at Univ. of Florida..

=== Constrained Cell Relaxation ===
Likewise, in all binaries above build 5.4.1.3-8-* the relaxation of the unit cell may be constrained
along one or more axes by creating a supplementary input file <code>CELL_RELAX</code>.
The subroutine <code>CONSTR_CELL_RELAX</code>,
having been a placeholder in the original source files,
was extended to read (and re-read) this file in each call and apply the containts
to the force components of each basis vector.
The performance impact from the multiple reads is expected to be minimal.

This is a Carbon-specific extension.

; File format:
cell relaxation - fixed string, for validation only, leading spaces ignored.
ix iy iz - 3 integer flags, indicating which cartesian components of the
basis vectors are allowed to be changed; 0 means no (fixed),
non-zero means yes (can change).
; Examples:
* To relax the entire cell (identical to default behavior):
cell relaxation
1 1 1
* To relax along z only:
cell relaxation
0 0 1
; Output indicator:
To indicate if your chosen VASP binary (a) looked for and (b) did or did not find the <code>CELL_RELAX</code> file, the subroutine <code>CONSTR_CELL_RELAX</code> will output one of the following each time it has been called:
CELL_RELAX not present: performing unconstrained unit cell relaxation
: or:
CELL_RELAX: performing constrained unit cell relaxation in cartesian directions 1 0 1
Unit cell forces:
1.1111111111111112 0.0000000000000000 2.2222222222222223
0.0000000000000000 0.0000000000000000 0.0000000000000000
2.2222222222222223 0.0000000000000000 3.3333333333333330
: (The force values shown here are contrived.)
; Author: Michael Sternberg.

== Usage under ASE ==
VASP calculations can be set up, run, and analyzed using the [https://wiki.fysik.dtu.dk/ase/ase/calculators/vasp.html calculator interface] of the [https://wiki.fysik.dtu.dk/ase/overview.html ASE] package, available on Carbon under both the [[../#ase3 | ase3]] and [[../#atk | atk]] modules.
ASE is told how to use VASP by means of environment variables which the modules define for you with sensible defaults:
VASP_PP_PATH
VASP_COMMAND
* The calculator will create and possibly overwrite VASP input files in the current directory, and then run VASP therein. To minimize clutter, consider setting up and changing into a run-specific directory before initializing the calculator.
* <code>VASP_COMMAND</code> normally uses the <code>vasp</code> binary ([[#Binaries|see above]]), and will run in parallel when called under a PBS job. That decision is based on the <code>PBS_NODEFILE</code> environment variable being present.
* You can supply another executable name or path in the variable <code>VASP_BINARY</code>. 
* The ASE calculator supports an alternative calling interface, a python script named in <code>VASP_SCRIPT</code>. This variable is not set by default on Carbon. If you wish to define a personal <code>VASP_SCRIPT</code>, unset <code>VASP_COMMAND</code> in tandem.
** <code>VASP_COMMAND</code> takes precendence over <code>VASP_SCRIPT</code>.
** The stdout stream is redirected to a file under <code>VASP_COMMAND</code>, but will appear on ASE's stdout under <code>VASP_SCRIPT</code>.
** Inspect the [https://trac.fysik.dtu.dk/projects/ase/browser/trunk/ase/calculators/vasp.py#L519 calculator source code] for details; see function <code>run(self)</code>.
* The directory for pseudopotentials is preconfigured to be the same as <code>$VASP_PP_PATH</code> for the ASE calculator.



== Node resources for VASP ==
Consider the follwing to specify PBS node resources (in job files and on the qsub command line).

=== Single-node jobs ===
* When memory-constrained, set <code>ppn</code> to less than the number of cores, and disallow node sharing:
#PBS -l nodes=1:ppn=16:gen6
#PBS -l naccesspolicy=SINGLEJOB -n

=== Multi-node jobs ===
Run on gen8, by using the following options for qsub or #PBS:
#PBS -l nodes=__:ppn=__:gen8

=== See also ===
* [[HPC/Submitting and Managing Jobs/Advanced node selection|Advanced node selection]]

HPC/Applications/vasp

2026-07-28T20:03:49Z

Stern: /* Requesting Access */

== Requesting Access ==
VASP versions 4, 5, and 6 are is available on Carbon, subject to confirmation for your group having a license for VASP major versions 4, 5, or 6.

To use VASP binaries on Carbon, send me (M. Sternberg), the following for the persons on your VASP license group who will be using VASP on Carbon:

* For each person:
** Full name
** Email address that the person registered with at vasp.at or MDI.



== Module considerations ==
* Review the available modules for each major version:
** [[HPC/Software/Modules#vasp | vasp]], for VASP 4.x
** [[HPC/Software/Modules#vasp5 | vasp5]], for VASP 5.x
** [[HPC/Software/Modules#vasp5 | vasp6]], for VASP 6.x
* Choose and [[HPC/Software/Module Setup#Shell Customizations|add '''one''' of the following lines to your ~/.bashrc]] file:
module load --auto vasp # sets '''$VASP_HOME'''

module load --auto vasp5 # sets '''$VASP5_HOME'''
module load --auto vasp6 # sets '''$VASP6_HOME'''
: Each module points to an installation directory using an [[HPC/Software/Module Setup#Package home directories|environment variable corresponding to the module name]]. Do not hardcode the directory anywhere, as that location is determined by the module and may change in subsequent versions. There should not be a reason to do so, since <code>$PATH</code> will be managed appropriately.

* Each module provides several binaries (see [[#Binaries|next section]]).
* You do not need to specify a path for vasp for the mpirun statement in the PBS job script.
* The modules are mutually exclusive.
* Potentials are available in <code>/opt/soft/vasp-pot</code>.

To access Graeme Henkelman's script suite, load the [[HPC/Software/Modules#vasp-vtstscripts|vasp-vtstscripts module]]. It works with both the vasp and vasp5 modules above. In either case, choose one of the VTST-patched binaries from the table below. The unpatched vasp binaries will not interoperate with the script suite.

module load vasp-vtstscripts

== Binaries ==
Some options for VASP cannot be configured at runtime (by input data) but must be chosen when compiling.
Since no single set of compile-time choices is appropriate for all uses, a number of binaries are provided, distinguished by their name.

The modules from versions vasp/4.6.35-mkl-13 and vasp5/5.2.12a-mkl-4 onwards, respectively,
contain several binaries built using different sets of preprocessor flags in the Makefile.
Earlier modules only contain two binaries, <code>vasp</code> and <code>vasp-gamma</code>.
Additional binaries are provided for useful preprocessor flag combinations
after applying patches from the group of Graeme Henkelman (UT Austin) to the source code:
* [http://theory.cm.utexas.edu/vtsttools/ Transition state tools for VASP], and
* [http://theory.cm.utexas.edu/vtsttools/bader/ DOS Projection in Bader Volumes].

To verify the binaries available for a module, run the appropriate command, once the module has been loaded:
ls $VASP_HOME/bin # for VASP-4 (previous major version)
ls $VASP5_HOME/bin

Choose a binary appropriate for your task and pass its name to <code>mpirun</code> in the job script.
There is no need for full path names.
Copy a sample job script from:
$VASP_HOME/vasp.job # for VASP-4 (previous major version)
$VASP5_HOME/vasp.job

=== Available binaries and their compile-time flags used ===

This table lists the names of the binaries compiled from
* combinations of preprocessor flags, and
* variants of the source code, primarily enhancing capabilities of ionic updates.

{| class="wikitable" cellpadding="5" style="text-align:left; font-family: monospace;"

|- style="background:#e4e4e4; text-align:center; font-weight: bold; font-family: sans-serif;"
| rowspan="2" | Versions and capabilities
| rowspan="2" | Flags used
| colspan="4" align="center" | Binary names
|- style="background:#e4e4e4; text-align:center; vertical-align:top; font-weight: bold; font-family: sans-serif;"
| Official
| Constrained Cell relaxation1
| Patched for VTST2
| Patched for Bader-projected DOS3
|-
! colspan="6" style="background:#e4e4e4; text-align:center; font-family: sans-serif;" |
|-

! rowspan="4" style="background:#e4e4e4; text-align:center; font-family: sans-serif;" | Conventional, Versions 4.x–5.3
| style="background:#e4e4e4;" | (none)
| style="background:none;" | '''vasp'''
| style="background:#eed;" | '''vasp-cellz'''
| style="background:#ded;" | '''vasp-vtst'''
| style="background:#fee;" | '''vasp-baderdos'''
|-
| style="background:#e4e4e4;" | -DNGZhalf
| style="background:none;" | '''vasp-chargehalf'''
| style="background:#eed;" | '''vasp-cellz-chargehalf'''
| style="background:#ded;" | '''vasp-vtst-chargehalf'''
| style="background:#fee;" | '''vasp-baderdos-chargehalf'''
|-
| style="background:#e4e4e4;" | -DwNGZhalf
| style="background:none;" | '''vasp-gamma'''
| style="background:#eed;" | '''vasp-cellz-gamma'''
| style="background:#ded;" | '''vasp-vtst-gamma'''
| style="background:#fee;" | '''vasp-baderdos-gamma'''
|-
| style="background:#e4e4e4;" | -DwNGZhalf -DNGZhalf
| style="background:none;" | '''vasp-gamma-chargehalf'''
| style="background:#eed;" | '''vasp-cellz-gamma-chargehalf'''
| style="background:#ded;" | '''vasp-vtst-gamma-chargehalf'''
| style="background:#fee;" | '''vasp-baderdos-gamma-chargehalf'''
|-
! rowspan="4" style="background:#e4e4e4; text-align:center; font-family: sans-serif;" | Advanced MD techniques,4 Versions 5.2 and 5.3
| style="background:#e4e4e4;" | -Dtbdyn
| style="background:#ddf;" | '''vasp-tbdyn'''
| rowspan="4" style="background:#f0f0f0; text-align:center; font-style:oblique; font-family: sans-serif;" | –
| rowspan="4" style="background:#f0f0f0; text-align:center; font-style:oblique; font-family: sans-serif;" | –
| rowspan="4" style="background:#f0f0f0; text-align:center; font-style:oblique; font-family: sans-serif;" | not useful
|-
| style="background:#e4e4e4;" | -Dtbdyn -DNGZhalf
| style="background:#ddf;" | '''vasp-tbdyn-chargehalf'''
|-
| style="background:#e4e4e4;" | -Dtbdyn -DwNGZhalf
| style="background:#ddf;" | '''vasp-tbdyn-gamma'''
|-
| style="background:#e4e4e4;" | -Dtbdyn -DwNGZhalf -DNGZhalf
| style="background:#ddf;" | '''vasp-tbdyn-gamma-chargehalf'''

|-
! colspan="6" style="background:#e4e4e4; text-align:center; font-family: sans-serif;" |
|-
! rowspan="5" style="background:#e4e4e4; text-align:center; font-family: sans-serif;" | Version 5.44,5
| style="background:#e4e4e4;" | -Dtbdyn
| style="background:#f0f0f0;" | '''vasp_ncl'''
| rowspan="5" style="background:#eed; text-align:center; font-family: sans-serif; " | (–o–)5
| style="background:#ded;" | '''vasp_vtst_ncl'''
| rowspan="5" style="background:#f0f0f0; text-align:center; font-style:oblique; font-family: sans-serif;" | TBD
|-
| style="background:#e4e4e4;" | -Dtbdyn -DNGZhalf
| style="background:none;" | '''vasp_std'''
| style="background:#ded;" | '''vasp_vtst_std'''
|-
| style="background:#e4e4e4;" | -Dtbdyn -DwNGZhalf -DNGZhalf
| style="background:none;" | '''vasp_gam'''
| style="background:#ded;" | '''vasp_vtst_gam'''
|-
| style="background:#e4e4e4;" | (GPU6)
| style="background:none;" | '''vasp_gpu_ncl'''
| style="background:#ded;" | '''vasp_vtst_gpu_ncl'''
|-
| style="background:#e4e4e4;" | (GPU6) -DNGZhalf
| style="background:none;" | '''vasp_gpu'''
| style="background:#ded;" | '''vasp_vtst_gpu'''

|-
! colspan="6" style="background:#e4e4e4; text-align:center; font-family: sans-serif;" |
|-
! rowspan="4" style="background:#e4e4e4; text-align:center; font-family: sans-serif;" | Version 6 (prelim.)
| style="background:#e4e4e4;" |
| style="background:#f0f0f0;" | '''vasp_ncl'''
| rowspan="3" style="background:#eed; text-align:center; font-family: sans-serif; " | (–o–)5
| style="background:#ded;" | '''vasp_vtst_ncl'''
| rowspan="4" style="background:#f0f0f0; text-align:center; font-style:oblique; font-family: sans-serif;" | TBD
|-
| style="background:#e4e4e4;" |
| style="background:none;" | '''vasp_std'''
| style="background:#ded;" | '''vasp_vtst_std'''
|-
| style="background:#e4e4e4;" |
| style="background:none;" | '''vasp_gam'''
| style="background:#ded;" | '''vasp_vtst_gam'''
|}

1) Stress tensor selectively zeroed in subroutine <code>CONSTR_CELL_RELAX</code> 
2) http://theory.cm.utexas.edu/vtsttools/ . The dimer method was broken for vasp5/5.3.3 prior to 5.3.3p3-mkl-2. 
3) http://theory.cm.utexas.edu/vtsttools/dos/ (now defunct) 
4) http://cms.mpi.univie.ac.at/vasp/vasp/Advanced_MD_techniques.html 
5) [[#Constrained Cell Relaxation|Axis-selective constrained cell relaxation]], controlled via an optional input file <code>CELL_RELAX</code>, is compiled into ''all'' binaries. 
6) http://cms.mpi.univie.ac.at/wiki/index.php/GPU_port_of_VASP 

=== Explanation of flags ===
* File <code>vasp.5.2/makefile.linux_pg</code> in the souce code explains:
<pre>
#-----------------------------------------------------------------------
# possible options for CPP:
# NGXhalf charge density reduced in X direction
# wNGXhalf gamma point only reduced in X direction
# avoidalloc avoid ALLOCATE if possible
# IFC work around some IFC bugs
# CACHE_SIZE 1000 for PII,PIII, 5000 for Athlon, 8000 P4
# RPROMU_DGEMV use DGEMV instead of DGEMM in RPRO (usually faster)
# RACCMU_DGEMV use DGEMV instead of DGEMM in RACC (faster on P4)
# **** definitely use -DRACCMU_DGEMV if you use the mkl library
#-----------------------------------------------------------------------
…
#-----------------------------------------------------------------------
# additional options for CPP in parallel version (see also above):
# NGZhalf charge density reduced in Z direction
# wNGZhalf gamma point only reduced in Z direction
# scaLAPACK use scaLAPACK (usually slower on 100 Mbit Net)
#-----------------------------------------------------------------------
</pre>
* In the new build system (since version 5.4) the direct use of the <code>-D*NG*half</code> options has been replaced by predefined make(1) arguments <code>VERSION={std,gam,ncl}</code>.

=== Common flags ===
-DIFC -DRPROMU_DGEMV -DRACCMU_DGEMV -DCACHE_SIZE=16000 -DMPI -DMPI_BLOCK=8000

== General patches ==

The following modifications have been applied and are active in ''all binaries'':

=== vdW-DF functional ===
The [http://cms.mpi.univie.ac.at/vasp/vasp/vdW_DF_functional_Langreth_Lundqvist_et_al.html vdW-DF functional of Langreth and Lundqvist et al.]
is implemented in all the <code>*-chargehalf</code> binaries of the vasp5 module, and all versions above 5.4.

No specific preparations are needed other than setting the INCAR flags as documented. The binaries will find the required precalculated kernel file <code>vdw_kernel.bindat</code> by means of the environment variable <code>$VASP5_HOME</code> which is set by the module.

This is a Carbon-specific extension.

=== Solvation Model ===
All binaries above build 5.4.1.3-8-* contain the [http://vaspsol.mse.ufl.edu VASPsol] solvation model of the Hennig group at Univ. of Florida..

=== Constrained Cell Relaxation ===
Likewise, in all binaries above build 5.4.1.3-8-* the relaxation of the unit cell may be constrained
along one or more axes by creating a supplementary input file <code>CELL_RELAX</code>.
The subroutine <code>CONSTR_CELL_RELAX</code>,
having been a placeholder in the original source files,
was extended to read (and re-read) this file in each call and apply the containts
to the force components of each basis vector.
The performance impact from the multiple reads is expected to be minimal.

This is a Carbon-specific extension.

; File format:
cell relaxation - fixed string, for validation only, leading spaces ignored.
ix iy iz - 3 integer flags, indicating which cartesian components of the
basis vectors are allowed to be changed; 0 means no (fixed),
non-zero means yes (can change).
; Examples:
* To relax the entire cell (identical to default behavior):
cell relaxation
1 1 1
* To relax along z only:
cell relaxation
0 0 1
; Output indicator:
To indicate if your chosen VASP binary (a) looked for and (b) did or did not find the <code>CELL_RELAX</code> file, the subroutine <code>CONSTR_CELL_RELAX</code> will output one of the following each time it has been called:
CELL_RELAX not present: performing unconstrained unit cell relaxation
: or:
CELL_RELAX: performing constrained unit cell relaxation in cartesian directions 1 0 1
Unit cell forces:
1.1111111111111112 0.0000000000000000 2.2222222222222223
0.0000000000000000 0.0000000000000000 0.0000000000000000
2.2222222222222223 0.0000000000000000 3.3333333333333330
: (The force values shown here are contrived.)
; Author: Michael Sternberg.

== Usage under ASE ==
VASP calculations can be set up, run, and analyzed using the [https://wiki.fysik.dtu.dk/ase/ase/calculators/vasp.html calculator interface] of the [https://wiki.fysik.dtu.dk/ase/overview.html ASE] package, available on Carbon under both the [[../#ase3 | ase3]] and [[../#atk | atk]] modules.
ASE is told how to use VASP by means of environment variables which the modules define for you with sensible defaults:
VASP_PP_PATH
VASP_COMMAND
* The calculator will create and possibly overwrite VASP input files in the current directory, and then run VASP therein. To minimize clutter, consider setting up and changing into a run-specific directory before initializing the calculator.
* <code>VASP_COMMAND</code> normally uses the <code>vasp</code> binary ([[#Binaries|see above]]), and will run in parallel when called under a PBS job. That decision is based on the <code>PBS_NODEFILE</code> environment variable being present.
* You can supply another executable name or path in the variable <code>VASP_BINARY</code>. 
* The ASE calculator supports an alternative calling interface, a python script named in <code>VASP_SCRIPT</code>. This variable is not set by default on Carbon. If you wish to define a personal <code>VASP_SCRIPT</code>, unset <code>VASP_COMMAND</code> in tandem.
** <code>VASP_COMMAND</code> takes precendence over <code>VASP_SCRIPT</code>.
** The stdout stream is redirected to a file under <code>VASP_COMMAND</code>, but will appear on ASE's stdout under <code>VASP_SCRIPT</code>.
** Inspect the [https://trac.fysik.dtu.dk/projects/ase/browser/trunk/ase/calculators/vasp.py#L519 calculator source code] for details; see function <code>run(self)</code>.
* The directory for pseudopotentials is preconfigured to be the same as <code>$VASP_PP_PATH</code> for the ASE calculator.



== Node resources for VASP ==
Consider the follwing to specify PBS node resources (in job files and on the qsub command line).

=== Single-node jobs ===
* When memory-constrained, set <code>ppn</code> to less than the number of cores, and disallow node sharing:
#PBS -l nodes=1:ppn=16:gen6
#PBS -l naccesspolicy=SINGLEJOB -n

=== Multi-node jobs ===
Run on gen8, by using the following options for qsub or #PBS:
#PBS -l nodes=__:ppn=__:gen8

=== See also ===
* [[HPC/Submitting and Managing Jobs/Advanced node selection|Advanced node selection]]

HPC/Applications/comsol

2026-06-09T17:52:57Z

Stern: /* Access */

== Introduction ==
The [http://www.comsol.com/products/multiphysics/ '''COMSOL Multiphysics'''] application suite
is available for Mac, Windows, and Linux.

== Access ==
[[Image:HPC Comsol licensed modules.png|right|450px|thumb|COMSOL features licensed at CNM as of 2026-06.]]
=== COMSOL Multiphysics ===
Access to '''COMSOL Multiphysics''', the main application with both a graphical user interface and batch computing capabilities, will be granted '''''upon request''''' .

You must be:
* An '''''Argonne employee'''''  in one of the following roles:
** '''''NST staff member'''''  (including PostDocs, joint appointments, and students at NST).
** '''''CNM Facility User'''''  (including ''non-NST Argonne employees''), with an active proposal that has COMSOL in scope. Please see [[HPC/FAQ#Access_and_proposals]].

The set of available COMSOL modules that we have licensed is shown at right.

* The native desktop version can be installed only on ''Argonne-owned'' computers (laptop or desktop). Please contact [mailto:help@anl.gov?Subject=COMSOL%20license%20access%20request CNM support]. Make sure to review [http://www.comsol.com/support/updates/ Product Updates].
* To run COMSOL on our HPC system ''Carbon'', as GUI or as batch computing jobs, your proposal must have ''Carbon'' access in scope.

=== COMSOL Server ===
''CNM Facility Users'' who are not Argonne employees are not eligible to access
the full '''COMSOL Multiphysics''' product, in compliance with its licensing terms and conditions.
However, the [https://www.comsol.com/comsol-server '''COMSOL Server'''] product offers an alternative
albeit more narrow means of access.

All users, including '''''non-Argonne users''''' may be able to run, over a web interface in their browser,
pre-parameterized "fixed-model" simulations that were made with the '''COMSOL Application Builder''' (requires Windows).

To inquire about using COMSOL server applications,
please contact us.

We will review possible purchase paths for COMSOL Server, and the ''considerable effort'' likely required by users and the CNM to develop and deploy a COMSOL application.

== Support ==
* For questions on how to run or start COMSOL:
*: Submit an [[HPC/Support | '''IT support request''']]
* For questions related to most COMSOL data or scripts, contact COMSOL support at:
*: [mailto:support@comsol.com '''support'''@comsol.com]
* '''Exception:''' If your support question might contain data that is subject to '''US Export Control''' regulations, contact instead:
*: [mailto:us-export-control@us.comsol.com '''us-export-control'''@'''us'''.comsol.com]

== The modes to run COMSOL ==
There are several ways of running COMSOL.

=== Running the desktop application ===
To edit models and analyze results, use the COMSOL graphical user interface (GUI). You can run it in several ways, each explained in more detail in the following sections.

# Run on your ''Argonne-provided'' computer, checking out Carbon's license remotely.
#* Requires a full installation of COMSOL on your desktop or laptop computer.
#* You will require access to the COMSOL license server running on Carbon, either through VPN (fairly easy) or SSH tunneling (can be intimidating for first-time users). See below for details.
# Run on Carbon, display on your desktop using the [http://en.wikipedia.org/wiki/X_Window_System X Window system (X11)].
#* Requires an ''X11 Server'' application running on your client machine, and SSH X11 forwarding.
#* The X11 protocol can be slow over LAN.
# Run on Carbon in a [[HPC/Network Access/Virtual Desktop | virtual desktop environment (VNC)]], viewed in a VNC client on your machine.
#* Requires a VNC client on your machine.
#* It can be confusing where VNC and the app runs.
#* The virtual desktop presents a (deliberatly) sparse windowing environment – Copy&Paste can be difficult to use.

=== Running a batch calculation ===
When the computational stage of models takes longer than a few minutes to complete, it will be beneficial to submit save the model file and run it in parallel on Carbon.
See the [[#Batch jobs|section below]].

== Native desktop version ==
To get the fastest response from the COMSOL GUI, run a native COMSOL version on your desktop machine or laptop, called ''client machine'' below.

Enter license details either in the COMSOL Setup configuration dialog.

== Use with X11 ==
For casual use, and with [[HPC/Network Access/SSH Tunnel Setup on Linux and MacOS | X11 tunneling to Carbon]] set up, simply add to your <code>~/.bashrc</code> file on Carbon:
module load comsol

Then, from the command line on a Carbon login node, start the COMSOL GUI:
comsol &

License configuration is automatic on Carbon.

Comsol will use a number of cores to run calculations multithreaded by default. (On Carbon, COMSOL_NUM_THREADS is set.)

'''Quit the application''' as soon as you are done. Only one user can use Comsol at a time.

== Use with VNC ==
See general [[HPC/Network Access/Virtual Desktop | instructions on using VNC]], and the COMSOL example therein.

== License configuration ==
A client's COMSOL license needs to be configured depending on how the client reaches CNM network resources. Follow the appropriate section:

* [[#License configuration when inside CNM or on CNM VPN|License configuration when '''inside CNM''' or on '''CNM VPN''']]
: Requires that the user's machine is logically within the network of CNM/NST, either by a wired/auth WiFi connection in the building or by VPN under a CNM/NST profile.
* [[#License configuration when outside CNM – requires SSH Tunneling|License configuration when '''outside CNM''' – requires SSH Tunneling]]
: ''Non-NST staff'' usually have a VPN profile that places the machine logically [[#License configuration when outside CNM – requires SSH Tunneling|outside of NST's network.]]

== License configuration when inside CNM or on CNM VPN ==
=== Ensure correct DNS search domains ===
* Inspect the client machine's network configuration.
* Ensure that '''two''' entries are listed as DNS Search Domains for the current network profile:
nst.anl.gov
cnm.anl.gov
: (The order of entries does not matter for the purposes considered here.)
* If the client machine is ''wired to CNM networks'', the domains are normally provided correctly by DHCP.
* If the client machine is ''connected over VPN'', you may need edit the respective network profile.
:* On a Mac, find the VPN configuration details under System Preferences, Network, Advanced… . You may need to unlock the Network panel first before you can make changes under "Advanced…". It appears that those changes are being saved and will be activated automatically the next time that this VPN session gets started.
: [[Image:VNC_add_cnm_to_DNS_1.png|400px]] [[Image:VNC_add_cnm_to_DNS_2.png|400px]]
:* On Windows or Linux machines, take ''analogous'' steps, using the platform's own configuration tools.

* Exclude license manager components from client desktop installation. (Those are all handled server-side.)

=== License format ===
* Floating Network License (FNL), "Three-server redundancy"

=== License servers ===
* On client machines, while running the COMSOL installer or license configuration, specify license servers and ports as follows:
1718 clicense1
1718 clicense2
1718 clicense3
: '''Note:''' Use ''short host names only''. (Technical Reason: The license must be usable both on Carbon's internal network as well as on divisional networks).
1718 clicense1<strike>.cnm.anl.gov</strike>
1718 clicense2<strike>.cnm.anl.gov</strike>
1718 clicense3<strike>.cnm.anl.gov</strike>

'''Advanced:''' A client machine's COMSOL license file <code><COMSOLDIR>/Multiphysics/license/license.dat</code> should look as follows:
<source lang="make">
SERVER clicense1 ANY 1718
SERVER clicense2 ANY 1718
SERVER clicense3 ANY 1718
USE_SERVER
</source>

<code><COMSOLDIR></code> is located by default at:
* '''Windows''': C:\Program files\COMSOL\COMSOL''nn''
* '''Linux''': /usr/local/comsol''nn''
* '''Mac''': /Applications/COMSOL''nn''
("''nn''" is the application's abbreviated version number.)

=== Network Troubleshooting ===
When using the license access model descibed here (as opposed to tunneled over SSH), ensure the following:
* Verify that short host names can be resolved to IP addresses. Either one of the following commands should produce sensible output:
host clicense1
: or:
ping -c 3 clicense1
* Verify that ''both license ports on the same server'' can be reached from your client machine:
telnet '''clicense1''' 1718
telnet '''clicense1''' 1719
: or:
telnet '''clicense2''' 1718
telnet '''clicense2''' 1719
: A ''successful'' connection looks like:
Trying 146.139.x.y...
Connected to clicense1.cnm.anl.gov.
Escape character is '^]'.
:: Do not enter any data. Instead, close each connection by hitting <code>Ctrl</code><code>]</code> (closing square bracket), followed by <code>Ctrl</code><code>D</code>.
: A ''failed'' connection gives one of these errors:
telnet: connect to address 146.139.x.y: Operation timed out
: or:
telnet: connect to address 146.139.x.y: Connection refused
:: In these cases, ask that your IP firewall configuration be reviewed and changed.

== License configuration when outside CNM – requires SSH Tunneling ==
This section documents how to run COMSOL from a network location outside CNM.

As a one-time preparation step, two items must be configured:
* The license host that COMSOL is to contact, and
* An SSH connection that will allow your machine to access the COMSOL license server inside CNM's network.

To then ''run'' the COMSOL application, the configured SSH connection must be up and kept running for the entire time that you run the COMSOL. When you exit COMSOL and wish to start it again, you can keep the SSH connection running, it will be re-used for the next COMSOL run.

=== Configure COMSOL ===
The following needs to be done just once, typically during COMSOL installation:

In the COMSOL Installer, or in COMSOL Preferences, enter:
* '''License format:''' Floating Network License (FNL), "Port number and Hostname"
:: Do ''not use'' "Three-server redundancy", since it cannot (easily) be accessed over SSH. Configure with just one server, and in the rare situation of a server failover, temporarily tweak your SSH configuration to contact the alternate server (see below).
* '''Server:''' <code>127.0.0.1</code>
:: Do not use "localhost", as COMSOL interprets this incorrectly to mean that your machine ''is'' a license server.

* '''Port:''' <code>1718</code>


=== Establish connection to CNM license server ===
This step needs to be done prior to running COMSOL in your desktop session:
* Connect to <code>mega</code> with the following command in a terminal window:
ssh -L 1718:'''clicense1''':1718 -L 1719:'''clicense1''':1719 mega.cnm.anl.gov
* You can leave this connection running in your terminal window across several invocations of COMSOL.

=== Run COMSOL ===
# Ensure your connection to "mega" is still running. If not, restart it.
# Start the COMSOL application.

=== Troubleshooting ===
* If you get an error that the number of licenses is "exhausted", that is confirmation that you correctly reached the license server, though other users are using licenses.
* If the license access attempt takes a long time or hangs, the service might temporarily run on a secondary server. In this case, stop the mega session and restart it, connecting to the alternate license server:
ssh -L 1718:clicense'''2''':1718 -L 1719:clicense'''2''':1719 mega.cnm.anl.gov

== General License Troubleshooting ==
The following are typical error messages.
=== Could not obtain license for COMSOL Multiphysics GUI ===
* Error details:
License error: -4.
'''Licensed number of users already reached.'''
Feature: COMSOLGUI
License path: /Users/stern/Applications/COMSOL52a/license/license.dat:
FlexNet Licensing error:-4,132
* Reason: The connection to the license server succeeded, but the license token(s) you requested are in use.
* Solution: Wait a bit, or [http://carbonmgmt/ganglia/lic.php identify who's using COMSOL] and politely ask the user when they might be finished with their work.

=== License server machine is down or not responding ===
* Error details:
License error: -96.
See the system administrator about starting the license server system, or
make sure you're referring to the right host (see LM_LICENSE_FILE).
Feature: SERIAL
Hostname: clicense3
License path: /Users/stern/Applications/COMSOL52a/license/license.dat:
FlexNet Licensing error:-96,7
* Reason: The license servers cannot be reached at all. The <code>ping</code> or <code>host</code> tests mentioned above fail.
* Solution: Review the client license file and DNS lookup domains.

=== License file does not support this version ===
* Error details:
Feature: SERIAL
'''Application version > License version: 5.21 > 3.5'''
License path: /Users/stern/Applications/COMSOL52a/license/license.dat:
FlexNet Licensing error:-21,126
* Reason: Only the first of two steps for connecting to the license servers succeeds, which may happen when the redundant license service runs on the alternate server instead of the one you're connected to.
* Solution: When using SSH, try <code>clicense2</code> instead of <code>clicense1</code>, or vice versa. Revisit your <code>~/.ssh/config</code> file or use the alternate command line as shown in the SSH section above.

== Multiple CPU cores ==
Ways to start multi-core (from http://www.comsol.com/support/knowledgebase/1096/)
comsol -np 8

export COMSOL_NUM_THREADS=8 # automatically provided on ''Carbon''
comsol
* COMSOL does not benefit from hyperthreading (oversubscribing the physical core count).

== Batch jobs ==

* http://www.comsol.com/support/knowledgebase/1001/ .
* [[Media:HPC 2012-05-16 Comsol parallel usage.pdf]] (Modified from http://www.comsol.com/support/knowledgebase/1001/files/4027/cluster_install_linux_42a.pptx – Appendix slide, PBS)
* '''Note:''' MPD is not supported on Carbon (I patched the main comsol script), and no longer necessary at all in Comsol-4.3.

=== Sample Job file ===
See:
$COMSOL_HOME/sample.job

== Important COMSOL Support Articles ==
* Toplevel suport page: https://www.comsol.com/support

=== KB Articles ===
* [https://www.comsol.com/support/knowledgebase/1001 Running COMSOL® in parallel on clusters]
* [https://www.comsol.com/support/knowledgebase/1250 Running parametric sweeps, batch sweeps, and cluster sweeps from the command line]
* [https://www.comsol.com/support/knowledgebase/1255 Reducing the amount of solution data stored in a model]
* [https://www.comsol.com/support/knowledgebase/830 Handling large models]

=== Blog-style articles ===
* [https://www.comsol.com/blogs/the-power-of-the-batch-sweep/ The Power of the Batch Sweep]
* [https://www.comsol.com/blogs/how-to-run-simulations-in-batch-mode-from-the-command-line/ How to Run Simulations in Batch Mode from the Command Line]

File:HPC Comsol licensed modules.png

2026-06-09T17:52:27Z

Stern: Stern uploaded a new version of File:HPC Comsol licensed modules.png

HPC/Support

2026-03-09T17:03:32Z

Stern: /* Required information */

== Review the FAQ list ==
For frequently asked questions regarding the Carbon HPC system, review '''[[HPC/FAQ | Frequently Asked Questions]]''' to find answers.

== Compose a support request for CNM HPC==
If your question is not answered in the FAQ: 
* [https://argonne.servicenowservices.com/sp?id=sc_cat_item&sys_id=4e37eb5b13199f0016ff33228144b036 '''Submit a support request'''] – Log in with your regular Argonne username and password (same as on Carbon).
* Choose a suitable Request Type and provide the details as shown in the next section.


== Required information ==

Be as ''specific as possible'' in your free-form description or form input fields.

=== For all issues – local and remote ===
Provide the following:
* The '''command''' you were trying to run or the menu item you chose in a graphical user interface
* The '''exact error message''' you get. Copy & Paste the message text, or take a screenshot and attach it to your mail.
* The '''directory path''' and '''file name''' you were working on, if applicable.
*: It is generally not necessary to include in your message copies of files that already reside on Carbon.
* Did the same command or action '''work previously''', if ever?
* Which '''changes''', if any, did you make to your configuration or input files since the last time this worked?
* Fill in an appropriate email '''Subject''' or form '''Title'''.
*: Best do this ''after'' you typed and reviewed the main text of your request.
* To reproduce your issue, we may ask for a [[#Test directory]] – (see next section)

=== For issues with PBS jobs ===
In addition to the [[#For all issues|preceding section]], for questions concerning PBS jobs, provide:
* the '''working directory''' of the job
* the name of the '''job file''' used
* the '''job number''' (for queued or running jobs)

=== For remote access ===
When you cannot log in to mega or a ''Carbon'' login node, provide:
* The details in section [[#For all issues]],
plus the following:
* The '''hostname''' you ran the command on. It is often shown in the command prompt. (If the prompt shows <code>localhost</code> it is not specific enough – ask your local support person for help.)
* The '''hostname''' you're trying to ''reach.''
* The '''username''' you use to connect.
* The '''software''' name and version you use to connect (e.g. SSH, VNC, or a browser).
* Your '''operating system''' and version.

If you use ssh or scp, repeat your command in verbose mode by supplying the "-v" option twice:
<source lang="bash">
scp -v -v [… other options and arguments …]
ssh -v -v [… other options and arguments …]
</source>
: Copy and Paste the possibly multi-page output into your request.



== Test directory ==
We may ask you to prepare a ''test directory'', which should help to:
* Reproduce your problem.
* Diagnose your problem, especially when errors are subtle or difficult to reproduce, such as where your application runs but does not fail outright.
* Confirm resolution.

Prepare the test directory so that it contains ''only'' files to run your application, either interactively or in a single PBS job, and possibly a file with correct output.
; Preferred location:
* In or under a ''sibling'' directory of one of a your failed runs (as opposed to a ''child'' directory).
; Include:
* PBS job file, if applicable,
* Run-specific data and script files (files named in input data or hardcoded in the code, possibly as symbolic links),
* If available, a file with the ''expected'' (correct) output, but with a file name different from the one generated by the job, such as by adding <code>.ref</code>.

In the course of diagnostics, we will suitably duplicate your directory as needed.

== Receiving support ==
When you get our support response:
* Carefully read it.
* Follow all instructions and answer all questions.
* For account- and password-related issues: it may take several hours for changes to take effect. If your initial attempt fails, wait at least that long before retrying.

== Additional considerations ==
To help us diagnose a problem:
* Read [http://www.chiark.greenend.org.uk/~sgtatham/bugs.html How to Report Bugs Effectively], by [http://www.chiark.greenend.org.uk/~sgtatham/ Simon Tatham], programmer.
* For deeper reading, consider [http://www.catb.org/~esr/faqs/smart-questions.html How To Ask Questions The Smart Way], by [http://en.wikipedia.org/wiki/Eric_S._Raymond Eric S. Raymond], open source pioneer.

HPC/Support

2026-03-09T17:01:06Z

Stern: /* Test directory */

== Review the FAQ list ==
For frequently asked questions regarding the Carbon HPC system, review '''[[HPC/FAQ | Frequently Asked Questions]]''' to find answers.

== Compose a support request for CNM HPC==
If your question is not answered in the FAQ: 
* [https://argonne.servicenowservices.com/sp?id=sc_cat_item&sys_id=4e37eb5b13199f0016ff33228144b036 '''Submit a support request'''] – Log in with your regular Argonne username and password (same as on Carbon).
* Choose a suitable Request Type and provide the details as shown in the next section.


== Required information ==

Be as ''specific as possible'' in your free-form description or form input fields.

=== For all issues – local and remote ===
Provide the following:
* The '''command''' you were trying to run or the menu item you chose in a graphical user interface
* The '''exact error message''' you get. Copy & Paste the message text, or take a screenshot and attach it to your mail.
* The '''directory path''' and '''file name''' you were working on, if applicable.
*: It is generally not necessary to include in your message copies of files that already reside on Carbon.
* Did the same command or action '''work previously''', if ever?
* Which '''changes''', if any, did you make to your configuration or input files since the last time this worked?
* Fill in an appropriate email '''Subject''' or form '''Title'''.
*: Best do this ''after'' you typed and reviewed the main text of your request.

=== For issues with PBS jobs ===
In addition to the [[#For all issues|preceding section]], for questions concerning PBS jobs, provide:
* the '''working directory''' of the job
* the name of the '''job file''' used
* the '''job number''' (for queued or running jobs)

=== For remote access ===
When you cannot log in to mega or a ''Carbon'' login node, provide:
* The details in section [[#For all issues]],
plus the following:
* The '''hostname''' you ran the command on. It is often shown in the command prompt. (If the prompt shows <code>localhost</code> it is not specific enough – ask your local support person for help.)
* The '''hostname''' you're trying to ''reach.''
* The '''username''' you use to connect.
* The '''software''' name and version you use to connect (e.g. SSH, VNC, or a browser).
* Your '''operating system''' and version.

If you use ssh or scp, repeat your command in verbose mode by supplying the "-v" option twice:
<source lang="bash">
scp -v -v [… other options and arguments …]
ssh -v -v [… other options and arguments …]
</source>
: Copy and Paste the possibly multi-page output into your request.



* [#Test directory] (see next section)

== Test directory ==
We may ask you to prepare a ''test directory'', which should help to:
* Reproduce your problem.
* Diagnose your problem, especially when errors are subtle or difficult to reproduce, such as where your application runs but does not fail outright.
* Confirm resolution.

Prepare the test directory so that it contains ''only'' files to run your application, either interactively or in a single PBS job, and possibly a file with correct output.
; Preferred location:
* In or under a ''sibling'' directory of one of a your failed runs (as opposed to a ''child'' directory).
; Include:
* PBS job file, if applicable,
* Run-specific data and script files (files named in input data or hardcoded in the code, possibly as symbolic links),
* If available, a file with the ''expected'' (correct) output, but with a file name different from the one generated by the job, such as by adding <code>.ref</code>.

In the course of diagnostics, we will suitably duplicate your directory as needed.

== Receiving support ==
When you get our support response:
* Carefully read it.
* Follow all instructions and answer all questions.
* For account- and password-related issues: it may take several hours for changes to take effect. If your initial attempt fails, wait at least that long before retrying.

== Additional considerations ==
To help us diagnose a problem:
* Read [http://www.chiark.greenend.org.uk/~sgtatham/bugs.html How to Report Bugs Effectively], by [http://www.chiark.greenend.org.uk/~sgtatham/ Simon Tatham], programmer.
* For deeper reading, consider [http://www.catb.org/~esr/faqs/smart-questions.html How To Ask Questions The Smart Way], by [http://en.wikipedia.org/wiki/Eric_S._Raymond Eric S. Raymond], open source pioneer.

HPC/Applications/comsol

2026-03-04T18:33:18Z

Stern:

== Introduction ==
The [http://www.comsol.com/products/multiphysics/ '''COMSOL Multiphysics'''] application suite
is available for Mac, Windows, and Linux.

== Access ==
[[Image:HPC Comsol licensed modules.png|right|450px|thumb|COMSOL features licensed at CNM as of 2024-10.]]
=== COMSOL Multiphysics ===
Access to '''COMSOL Multiphysics''', the main application with both a graphical user interface and batch computing capabilities, will be granted '''''upon request''''' .

You must be:
* An '''''Argonne employee'''''  in one of the following roles:
** '''''NST staff member'''''  (including PostDocs, joint appointments, and students at NST).
** '''''CNM Facility User'''''  (including ''non-NST Argonne employees''), with an active proposal that has COMSOL in scope. Please see [[HPC/FAQ#Access_and_proposals]].

The set of available COMSOL modules that we have licensed is shown at right.

* The native desktop version can be installed only on ''Argonne-owned'' computers (laptop or desktop). Please contact [mailto:help@anl.gov?Subject=COMSOL%20license%20access%20request CNM support]. Make sure to review [http://www.comsol.com/support/updates/ Product Updates].
* To run COMSOL on our HPC system ''Carbon'', as GUI or as batch computing jobs, your proposal must have ''Carbon'' access in scope.

=== COMSOL Server ===
''CNM Facility Users'' who are not Argonne employees are not eligible to access
the full '''COMSOL Multiphysics''' product, in compliance with its licensing terms and conditions.
However, the [https://www.comsol.com/comsol-server '''COMSOL Server'''] product offers an alternative
albeit more narrow means of access.

All users, including '''''non-Argonne users''''' may be able to run, over a web interface in their browser,
pre-parameterized "fixed-model" simulations that were made with the '''COMSOL Application Builder''' (requires Windows).

To inquire about using COMSOL server applications,
please contact us.

We will review possible purchase paths for COMSOL Server, and the ''considerable effort'' likely required by users and the CNM to develop and deploy a COMSOL application.

== Support ==
* For questions on how to run or start COMSOL:
*: Submit an [[HPC/Support | '''IT support request''']]
* For questions related to most COMSOL data or scripts, contact COMSOL support at:
*: [mailto:support@comsol.com '''support'''@comsol.com]
* '''Exception:''' If your support question might contain data that is subject to '''US Export Control''' regulations, contact instead:
*: [mailto:us-export-control@us.comsol.com '''us-export-control'''@'''us'''.comsol.com]

== The modes to run COMSOL ==
There are several ways of running COMSOL.

=== Running the desktop application ===
To edit models and analyze results, use the COMSOL graphical user interface (GUI). You can run it in several ways, each explained in more detail in the following sections.

# Run on your ''Argonne-provided'' computer, checking out Carbon's license remotely.
#* Requires a full installation of COMSOL on your desktop or laptop computer.
#* You will require access to the COMSOL license server running on Carbon, either through VPN (fairly easy) or SSH tunneling (can be intimidating for first-time users). See below for details.
# Run on Carbon, display on your desktop using the [http://en.wikipedia.org/wiki/X_Window_System X Window system (X11)].
#* Requires an ''X11 Server'' application running on your client machine, and SSH X11 forwarding.
#* The X11 protocol can be slow over LAN.
# Run on Carbon in a [[HPC/Network Access/Virtual Desktop | virtual desktop environment (VNC)]], viewed in a VNC client on your machine.
#* Requires a VNC client on your machine.
#* It can be confusing where VNC and the app runs.
#* The virtual desktop presents a (deliberatly) sparse windowing environment – Copy&Paste can be difficult to use.

=== Running a batch calculation ===
When the computational stage of models takes longer than a few minutes to complete, it will be beneficial to submit save the model file and run it in parallel on Carbon.
See the [[#Batch jobs|section below]].

== Native desktop version ==
To get the fastest response from the COMSOL GUI, run a native COMSOL version on your desktop machine or laptop, called ''client machine'' below.

Enter license details either in the COMSOL Setup configuration dialog.

== Use with X11 ==
For casual use, and with [[HPC/Network Access/SSH Tunnel Setup on Linux and MacOS | X11 tunneling to Carbon]] set up, simply add to your <code>~/.bashrc</code> file on Carbon:
module load comsol

Then, from the command line on a Carbon login node, start the COMSOL GUI:
comsol &

License configuration is automatic on Carbon.

Comsol will use a number of cores to run calculations multithreaded by default. (On Carbon, COMSOL_NUM_THREADS is set.)

'''Quit the application''' as soon as you are done. Only one user can use Comsol at a time.

== Use with VNC ==
See general [[HPC/Network Access/Virtual Desktop | instructions on using VNC]], and the COMSOL example therein.

== License configuration ==
A client's COMSOL license needs to be configured depending on how the client reaches CNM network resources. Follow the appropriate section:

* [[#License configuration when inside CNM or on CNM VPN|License configuration when '''inside CNM''' or on '''CNM VPN''']]
: Requires that the user's machine is logically within the network of CNM/NST, either by a wired/auth WiFi connection in the building or by VPN under a CNM/NST profile.
* [[#License configuration when outside CNM – requires SSH Tunneling|License configuration when '''outside CNM''' – requires SSH Tunneling]]
: ''Non-NST staff'' usually have a VPN profile that places the machine logically [[#License configuration when outside CNM – requires SSH Tunneling|outside of NST's network.]]

== License configuration when inside CNM or on CNM VPN ==
=== Ensure correct DNS search domains ===
* Inspect the client machine's network configuration.
* Ensure that '''two''' entries are listed as DNS Search Domains for the current network profile:
nst.anl.gov
cnm.anl.gov
: (The order of entries does not matter for the purposes considered here.)
* If the client machine is ''wired to CNM networks'', the domains are normally provided correctly by DHCP.
* If the client machine is ''connected over VPN'', you may need edit the respective network profile.
:* On a Mac, find the VPN configuration details under System Preferences, Network, Advanced… . You may need to unlock the Network panel first before you can make changes under "Advanced…". It appears that those changes are being saved and will be activated automatically the next time that this VPN session gets started.
: [[Image:VNC_add_cnm_to_DNS_1.png|400px]] [[Image:VNC_add_cnm_to_DNS_2.png|400px]]
:* On Windows or Linux machines, take ''analogous'' steps, using the platform's own configuration tools.

* Exclude license manager components from client desktop installation. (Those are all handled server-side.)

=== License format ===
* Floating Network License (FNL), "Three-server redundancy"

=== License servers ===
* On client machines, while running the COMSOL installer or license configuration, specify license servers and ports as follows:
1718 clicense1
1718 clicense2
1718 clicense3
: '''Note:''' Use ''short host names only''. (Technical Reason: The license must be usable both on Carbon's internal network as well as on divisional networks).
1718 clicense1<strike>.cnm.anl.gov</strike>
1718 clicense2<strike>.cnm.anl.gov</strike>
1718 clicense3<strike>.cnm.anl.gov</strike>

'''Advanced:''' A client machine's COMSOL license file <code><COMSOLDIR>/Multiphysics/license/license.dat</code> should look as follows:
<source lang="make">
SERVER clicense1 ANY 1718
SERVER clicense2 ANY 1718
SERVER clicense3 ANY 1718
USE_SERVER
</source>

<code><COMSOLDIR></code> is located by default at:
* '''Windows''': C:\Program files\COMSOL\COMSOL''nn''
* '''Linux''': /usr/local/comsol''nn''
* '''Mac''': /Applications/COMSOL''nn''
("''nn''" is the application's abbreviated version number.)

=== Network Troubleshooting ===
When using the license access model descibed here (as opposed to tunneled over SSH), ensure the following:
* Verify that short host names can be resolved to IP addresses. Either one of the following commands should produce sensible output:
host clicense1
: or:
ping -c 3 clicense1
* Verify that ''both license ports on the same server'' can be reached from your client machine:
telnet '''clicense1''' 1718
telnet '''clicense1''' 1719
: or:
telnet '''clicense2''' 1718
telnet '''clicense2''' 1719
: A ''successful'' connection looks like:
Trying 146.139.x.y...
Connected to clicense1.cnm.anl.gov.
Escape character is '^]'.
:: Do not enter any data. Instead, close each connection by hitting <code>Ctrl</code><code>]</code> (closing square bracket), followed by <code>Ctrl</code><code>D</code>.
: A ''failed'' connection gives one of these errors:
telnet: connect to address 146.139.x.y: Operation timed out
: or:
telnet: connect to address 146.139.x.y: Connection refused
:: In these cases, ask that your IP firewall configuration be reviewed and changed.

== License configuration when outside CNM – requires SSH Tunneling ==
This section documents how to run COMSOL from a network location outside CNM.

As a one-time preparation step, two items must be configured:
* The license host that COMSOL is to contact, and
* An SSH connection that will allow your machine to access the COMSOL license server inside CNM's network.

To then ''run'' the COMSOL application, the configured SSH connection must be up and kept running for the entire time that you run the COMSOL. When you exit COMSOL and wish to start it again, you can keep the SSH connection running, it will be re-used for the next COMSOL run.

=== Configure COMSOL ===
The following needs to be done just once, typically during COMSOL installation:

In the COMSOL Installer, or in COMSOL Preferences, enter:
* '''License format:''' Floating Network License (FNL), "Port number and Hostname"
:: Do ''not use'' "Three-server redundancy", since it cannot (easily) be accessed over SSH. Configure with just one server, and in the rare situation of a server failover, temporarily tweak your SSH configuration to contact the alternate server (see below).
* '''Server:''' <code>127.0.0.1</code>
:: Do not use "localhost", as COMSOL interprets this incorrectly to mean that your machine ''is'' a license server.

* '''Port:''' <code>1718</code>


=== Establish connection to CNM license server ===
This step needs to be done prior to running COMSOL in your desktop session:
* Connect to <code>mega</code> with the following command in a terminal window:
ssh -L 1718:'''clicense1''':1718 -L 1719:'''clicense1''':1719 mega.cnm.anl.gov
* You can leave this connection running in your terminal window across several invocations of COMSOL.

=== Run COMSOL ===
# Ensure your connection to "mega" is still running. If not, restart it.
# Start the COMSOL application.

=== Troubleshooting ===
* If you get an error that the number of licenses is "exhausted", that is confirmation that you correctly reached the license server, though other users are using licenses.
* If the license access attempt takes a long time or hangs, the service might temporarily run on a secondary server. In this case, stop the mega session and restart it, connecting to the alternate license server:
ssh -L 1718:clicense'''2''':1718 -L 1719:clicense'''2''':1719 mega.cnm.anl.gov

== General License Troubleshooting ==
The following are typical error messages.
=== Could not obtain license for COMSOL Multiphysics GUI ===
* Error details:
License error: -4.
'''Licensed number of users already reached.'''
Feature: COMSOLGUI
License path: /Users/stern/Applications/COMSOL52a/license/license.dat:
FlexNet Licensing error:-4,132
* Reason: The connection to the license server succeeded, but the license token(s) you requested are in use.
* Solution: Wait a bit, or [http://carbonmgmt/ganglia/lic.php identify who's using COMSOL] and politely ask the user when they might be finished with their work.

=== License server machine is down or not responding ===
* Error details:
License error: -96.
See the system administrator about starting the license server system, or
make sure you're referring to the right host (see LM_LICENSE_FILE).
Feature: SERIAL
Hostname: clicense3
License path: /Users/stern/Applications/COMSOL52a/license/license.dat:
FlexNet Licensing error:-96,7
* Reason: The license servers cannot be reached at all. The <code>ping</code> or <code>host</code> tests mentioned above fail.
* Solution: Review the client license file and DNS lookup domains.

=== License file does not support this version ===
* Error details:
Feature: SERIAL
'''Application version > License version: 5.21 > 3.5'''
License path: /Users/stern/Applications/COMSOL52a/license/license.dat:
FlexNet Licensing error:-21,126
* Reason: Only the first of two steps for connecting to the license servers succeeds, which may happen when the redundant license service runs on the alternate server instead of the one you're connected to.
* Solution: When using SSH, try <code>clicense2</code> instead of <code>clicense1</code>, or vice versa. Revisit your <code>~/.ssh/config</code> file or use the alternate command line as shown in the SSH section above.

== Multiple CPU cores ==
Ways to start multi-core (from http://www.comsol.com/support/knowledgebase/1096/)
comsol -np 8

export COMSOL_NUM_THREADS=8 # automatically provided on ''Carbon''
comsol
* COMSOL does not benefit from hyperthreading (oversubscribing the physical core count).

== Batch jobs ==

* http://www.comsol.com/support/knowledgebase/1001/ .
* [[Media:HPC 2012-05-16 Comsol parallel usage.pdf]] (Modified from http://www.comsol.com/support/knowledgebase/1001/files/4027/cluster_install_linux_42a.pptx – Appendix slide, PBS)
* '''Note:''' MPD is not supported on Carbon (I patched the main comsol script), and no longer necessary at all in Comsol-4.3.

=== Sample Job file ===
See:
$COMSOL_HOME/sample.job

== Important COMSOL Support Articles ==
* Toplevel suport page: https://www.comsol.com/support

=== KB Articles ===
* [https://www.comsol.com/support/knowledgebase/1001 Running COMSOL® in parallel on clusters]
* [https://www.comsol.com/support/knowledgebase/1250 Running parametric sweeps, batch sweeps, and cluster sweeps from the command line]
* [https://www.comsol.com/support/knowledgebase/1255 Reducing the amount of solution data stored in a model]
* [https://www.comsol.com/support/knowledgebase/830 Handling large models]

=== Blog-style articles ===
* [https://www.comsol.com/blogs/the-power-of-the-batch-sweep/ The Power of the Batch Sweep]
* [https://www.comsol.com/blogs/how-to-run-simulations-in-batch-mode-from-the-command-line/ How to Run Simulations in Batch Mode from the Command Line]

HPC/Applications/comsol

2026-03-04T18:31:45Z

Stern:

== Introduction ==
The [http://www.comsol.com/products/multiphysics/ '''COMSOL Multiphysics'''] application suite
is available for Mac, Windows, and Linux.

== Access ==
[[Image:HPC Comsol licensed modules.png|right|450px|thumb|COMSOL features licensed at CNM as of 2024-10.]]
=== COMSOL Multiphysics ===
Access to '''COMSOL Multiphysics''', the main application with both a graphical user interface and batch computing capabilities, will be granted '''''upon request''''' .

You must be:
* An '''''Argonne employee'''''  in one of the following roles:
** '''''NST staff member'''''  (including PostDocs, joint appointments, and students at NST).
** '''''CNM Facility User'''''  (including ''non-NST Argonne employees''), with an active proposal that has COMSOL in scope. Please see [[HPC/FAQ#Access_and_proposals]].

The set of available COMSOL modules that we have licensed is shown at right.

* The native desktop version can be installed only on ''Argonne-owned'' computers (laptop or desktop). Please contact [mailto:help@anl.gov?Subject=COMSOL%20license%20access%20request CNM support]. Make sure to review [http://www.comsol.com/support/updates/ Product Updates].
* To run COMSOL on our HPC system ''Carbon'', as GUI or as batch computing jobs, your proposal must have ''Carbon'' access in scope.

=== COMSOL Server ===
''CNM Facility Users'' who are not Argonne employees are not eligible to access
the full '''COMSOL Multiphysics''' product, in compliance with its licensing terms and conditions.
However, the [https://www.comsol.com/comsol-server '''COMSOL Server'''] product offers an alternative
albeit more narrow means of access.

All users, including '''''non-Argonne users''''' may be able to run, over a web interface in their browser,
pre-parameterized "fixed-model" simulations that were made with the '''COMSOL Application Builder''' (requires Windows).

To inquire about using COMSOL server applications,
please contact us.

We will review possible purchase paths for COMSOL Server, and the ''considerable effort'' likely required by users and the CNM to develop and deploy a COMSOL application.

== Support ==
* For questions on how to run or start COMSOL:
*: Submit an [[HPC/Support | '''IT support request''']]
* For questions related to most COMSOL data or scripts, contact COMSOL support at:
*: [mailto:support@comsol.com '''support'''@comsol.com]
* '''Exception:''' If your support question might contain data that is subject to '''US Export Control''' regulations, contact instead:
*: [mailto:us-export-control@us.comsol.com '''us-export-control'''@'''us'''.comsol.com]

== The modes to run COMSOL ==
There are several ways of running COMSOL.

=== Running the desktop application ===
To edit models and analyze results, use the COMSOL graphical user interface (GUI). You can run it in several ways, each explained in more detail in the following sections.

# Run on your ''Argonne-provided'' computer, checking out Carbon's license remotely.
#* Requires a full installation of COMSOL on your desktop or laptop computer.
#* You will require access to the COMSOL license server running on Carbon, either through VPN (fairly easy) or SSH tunneling (can be intimidating for first-time users). See below for details.
# Run on Carbon, display on your desktop using the [http://en.wikipedia.org/wiki/X_Window_System X Window system (X11)].
#* Requires an ''X11 Server'' application running on your client machine, and SSH X11 forwarding.
#* The X11 protocol can be slow over LAN.
# Run on Carbon in a [[HPC/Network Access/Virtual Desktop | virtual desktop environment (VNC)]], viewed in a VNC client on your machine.
#* Requires a VNC client on your machine.
#* It can be confusing where VNC and the app runs.
#* The virtual desktop presents a (deliberatly) sparse windowing environment – Copy&Paste can be difficult to use.

=== Running a batch calculation ===
When the computational stage of models takes longer than a few minutes to complete, it will be beneficial to submit save the model file and run it in parallel on Carbon.
See the [[#Batch jobs|section below]].

== Use with X11 ==
For casual use, and with [[HPC/Network Access/SSH Tunnel Setup on Linux and MacOS | X11 tunneling to Carbon]] set up, simply add to your <code>~/.bashrc</code> file on Carbon:
module load comsol

Then, from the command line on a Carbon login node, start the COMSOL GUI:
comsol &

License configuration is automatic on Carbon.

Comsol will use a number of cores to run calculations multithreaded by default. (On Carbon, COMSOL_NUM_THREADS is set.)

'''Quit the application''' as soon as you are done. Only one user can use Comsol at a time.

== Use with VNC ==
See general [[HPC/Network Access/Virtual Desktop | instructions on using VNC]], and the COMSOL example therein.

== Native desktop version ==
To get the fastest response from the COMSOL GUI, run a native COMSOL version on your desktop machine or laptop, called ''client machine'' below.

Enter license details either in the COMSOL Setup configuration dialog.

== License configuration ==
A client's COMSOL license needs to be configured depending on how the client reaches CNM network resources. Follow the appropriate section:

* [[#License configuration when inside CNM or on CNM VPN|License configuration when '''inside CNM''' or on '''CNM VPN''']]
: Requires that the user's machine is logically within the network of CNM/NST, either by a wired/auth WiFi connection in the building or by VPN under a CNM/NST profile.
* [[#License configuration when outside CNM – requires SSH Tunneling|License configuration when '''outside CNM''' – requires SSH Tunneling]]
: ''Non-NST staff'' usually have a VPN profile that places the machine logically [[#License configuration when outside CNM – requires SSH Tunneling|outside of NST's network.]]

== License configuration when inside CNM or on CNM VPN ==
=== Ensure correct DNS search domains ===
* Inspect the client machine's network configuration.
* Ensure that '''two''' entries are listed as DNS Search Domains for the current network profile:
nst.anl.gov
cnm.anl.gov
: (The order of entries does not matter for the purposes considered here.)
* If the client machine is ''wired to CNM networks'', the domains are normally provided correctly by DHCP.
* If the client machine is ''connected over VPN'', you may need edit the respective network profile.
:* On a Mac, find the VPN configuration details under System Preferences, Network, Advanced… . You may need to unlock the Network panel first before you can make changes under "Advanced…". It appears that those changes are being saved and will be activated automatically the next time that this VPN session gets started.
: [[Image:VNC_add_cnm_to_DNS_1.png|400px]] [[Image:VNC_add_cnm_to_DNS_2.png|400px]]
:* On Windows or Linux machines, take ''analogous'' steps, using the platform's own configuration tools.

* Exclude license manager components from client desktop installation. (Those are all handled server-side.)

=== License format ===
* Floating Network License (FNL), "Three-server redundancy"

=== License servers ===
* On client machines, while running the COMSOL installer or license configuration, specify license servers and ports as follows:
1718 clicense1
1718 clicense2
1718 clicense3
: '''Note:''' Use ''short host names only''. (Technical Reason: The license must be usable both on Carbon's internal network as well as on divisional networks).
1718 clicense1<strike>.cnm.anl.gov</strike>
1718 clicense2<strike>.cnm.anl.gov</strike>
1718 clicense3<strike>.cnm.anl.gov</strike>

'''Advanced:''' A client machine's COMSOL license file <code><COMSOLDIR>/Multiphysics/license/license.dat</code> should look as follows:
<source lang="make">
SERVER clicense1 ANY 1718
SERVER clicense2 ANY 1718
SERVER clicense3 ANY 1718
USE_SERVER
</source>

<code><COMSOLDIR></code> is located by default at:
* '''Windows''': C:\Program files\COMSOL\COMSOL''nn''
* '''Linux''': /usr/local/comsol''nn''
* '''Mac''': /Applications/COMSOL''nn''
("''nn''" is the application's abbreviated version number.)

=== Network Troubleshooting ===
When using the license access model descibed here (as opposed to tunneled over SSH), ensure the following:
* Verify that short host names can be resolved to IP addresses. Either one of the following commands should produce sensible output:
host clicense1
: or:
ping -c 3 clicense1
* Verify that ''both license ports on the same server'' can be reached from your client machine:
telnet '''clicense1''' 1718
telnet '''clicense1''' 1719
: or:
telnet '''clicense2''' 1718
telnet '''clicense2''' 1719
: A ''successful'' connection looks like:
Trying 146.139.x.y...
Connected to clicense1.cnm.anl.gov.
Escape character is '^]'.
:: Do not enter any data. Instead, close each connection by hitting <code>Ctrl</code><code>]</code> (closing square bracket), followed by <code>Ctrl</code><code>D</code>.
: A ''failed'' connection gives one of these errors:
telnet: connect to address 146.139.x.y: Operation timed out
: or:
telnet: connect to address 146.139.x.y: Connection refused
:: In these cases, ask that your IP firewall configuration be reviewed and changed.

== License configuration when outside CNM – requires SSH Tunneling ==
This section documents how to run COMSOL from a network location outside CNM.

As a one-time preparation step, two items must be configured:
* The license host that COMSOL is to contact, and
* An SSH connection that will allow your machine to access the COMSOL license server inside CNM's network.

To then ''run'' the COMSOL application, the configured SSH connection must be up and kept running for the entire time that you run the COMSOL. When you exit COMSOL and wish to start it again, you can keep the SSH connection running, it will be re-used for the next COMSOL run.

=== Configure COMSOL ===
The following needs to be done just once, typically during COMSOL installation:

In the COMSOL Installer, or in COMSOL Preferences, enter:
* '''License format:''' Floating Network License (FNL), "Port number and Hostname"
:: Do ''not use'' "Three-server redundancy", since it cannot (easily) be accessed over SSH. Configure with just one server, and in the rare situation of a server failover, temporarily tweak your SSH configuration to contact the alternate server (see below).
* '''Server:''' <code>127.0.0.1</code>
:: Do not use "localhost", as COMSOL interprets this incorrectly to mean that your machine ''is'' a license server.

* '''Port:''' <code>1718</code>


=== Establish connection to CNM license server ===
This step needs to be done prior to running COMSOL in your desktop session:
* Connect to <code>mega</code> with the following command in a terminal window:
ssh -L 1718:'''clicense1''':1718 -L 1719:'''clicense1''':1719 mega.cnm.anl.gov
* You can leave this connection running in your terminal window across several invocations of COMSOL.

=== Run COMSOL ===
# Ensure your connection to "mega" is still running. If not, restart it.
# Start the COMSOL application.

=== Troubleshooting ===
* If you get an error that the number of licenses is "exhausted", that is confirmation that you correctly reached the license server, though other users are using licenses.
* If the license access attempt takes a long time or hangs, the service might temporarily run on a secondary server. In this case, stop the mega session and restart it, connecting to the alternate license server:
ssh -L 1718:clicense'''2''':1718 -L 1719:clicense'''2''':1719 mega.cnm.anl.gov

== General License Troubleshooting ==
The following are typical error messages.
=== Could not obtain license for COMSOL Multiphysics GUI ===
* Error details:
License error: -4.
'''Licensed number of users already reached.'''
Feature: COMSOLGUI
License path: /Users/stern/Applications/COMSOL52a/license/license.dat:
FlexNet Licensing error:-4,132
* Reason: The connection to the license server succeeded, but the license token(s) you requested are in use.
* Solution: Wait a bit, or [http://carbonmgmt/ganglia/lic.php identify who's using COMSOL] and politely ask the user when they might be finished with their work.

=== License server machine is down or not responding ===
* Error details:
License error: -96.
See the system administrator about starting the license server system, or
make sure you're referring to the right host (see LM_LICENSE_FILE).
Feature: SERIAL
Hostname: clicense3
License path: /Users/stern/Applications/COMSOL52a/license/license.dat:
FlexNet Licensing error:-96,7
* Reason: The license servers cannot be reached at all. The <code>ping</code> or <code>host</code> tests mentioned above fail.
* Solution: Review the client license file and DNS lookup domains.

=== License file does not support this version ===
* Error details:
Feature: SERIAL
'''Application version > License version: 5.21 > 3.5'''
License path: /Users/stern/Applications/COMSOL52a/license/license.dat:
FlexNet Licensing error:-21,126
* Reason: Only the first of two steps for connecting to the license servers succeeds, which may happen when the redundant license service runs on the alternate server instead of the one you're connected to.
* Solution: When using SSH, try <code>clicense2</code> instead of <code>clicense1</code>, or vice versa. Revisit your <code>~/.ssh/config</code> file or use the alternate command line as shown in the SSH section above.

== Multiple CPU cores ==
Ways to start multi-core (from http://www.comsol.com/support/knowledgebase/1096/)
comsol -np 8

export COMSOL_NUM_THREADS=8 # automatically provided on ''Carbon''
comsol
* COMSOL does not benefit from hyperthreading (oversubscribing the physical core count).

== Batch jobs ==

* http://www.comsol.com/support/knowledgebase/1001/ .
* [[Media:HPC 2012-05-16 Comsol parallel usage.pdf]] (Modified from http://www.comsol.com/support/knowledgebase/1001/files/4027/cluster_install_linux_42a.pptx – Appendix slide, PBS)
* '''Note:''' MPD is not supported on Carbon (I patched the main comsol script), and no longer necessary at all in Comsol-4.3.

=== Sample Job file ===
See:
$COMSOL_HOME/sample.job

== Important COMSOL Support Articles ==
* Toplevel suport page: https://www.comsol.com/support

=== KB Articles ===
* [https://www.comsol.com/support/knowledgebase/1001 Running COMSOL® in parallel on clusters]
* [https://www.comsol.com/support/knowledgebase/1250 Running parametric sweeps, batch sweeps, and cluster sweeps from the command line]
* [https://www.comsol.com/support/knowledgebase/1255 Reducing the amount of solution data stored in a model]
* [https://www.comsol.com/support/knowledgebase/830 Handling large models]

=== Blog-style articles ===
* [https://www.comsol.com/blogs/the-power-of-the-batch-sweep/ The Power of the Batch Sweep]
* [https://www.comsol.com/blogs/how-to-run-simulations-in-batch-mode-from-the-command-line/ How to Run Simulations in Batch Mode from the Command Line]

HPC/Applications/comsol

2026-03-04T18:30:28Z

Stern: roll back 2 revs.

== Introduction ==
The [http://www.comsol.com/products/multiphysics/ '''COMSOL Multiphysics'''] application suite
is available for Mac, Windows, and Linux.

== Access ==
[[Image:HPC Comsol licensed modules.png|right|450px|thumb|COMSOL features licensed at CNM as of 2024-10.]]
=== COMSOL Multiphysics ===
Access to '''COMSOL Multiphysics''', the main application with both a graphical user interface and batch computing capabilities, will be granted '''''upon request''''' .

You must be:
* An '''''Argonne employee'''''  in one of the following roles:
** '''''NST staff member'''''  (including PostDocs, joint appointments, and students at NST).
** '''''CNM Facility User'''''  (including ''non-NST Argonne employees''), with an active proposal that has COMSOL in scope. Please see [[HPC/FAQ#Access_and_proposals]].

The set of available COMSOL modules that we have licensed is shown at right.

* The native desktop version can be installed only on ''Argonne-owned'' computers (laptop or desktop). Please contact [mailto:help@anl.gov?Subject=COMSOL%20license%20access%20request CNM support]. Make sure to review [http://www.comsol.com/support/updates/ Product Updates].
* To run COMSOL on our HPC system ''Carbon'', as GUI or as batch computing jobs, your proposal must have ''Carbon'' access in scope.

=== COMSOL Server ===
''CNM Facility Users'' who are not Argonne employees are not eligible to access
the full '''COMSOL Multiphysics''' product, in compliance with its licensing terms and conditions.
However, the [https://www.comsol.com/comsol-server '''COMSOL Server'''] product offers an alternative
albeit more narrow means of access.

All users, including '''''non-Argonne users''''' may be able to run, over a web interface in their browser,
pre-parameterized "fixed-model" simulations that were made with the '''COMSOL Application Builder''' (requires Windows).

To inquire about using COMSOL server applications,
please contact us.

We will review possible purchase paths for COMSOL Server, and the ''considerable effort'' likely required by users and the CNM to develop and deploy a COMSOL application.

== Support ==
* For questions on how to run or start COMSOL:
*: Submit an [[HPC/Support | '''IT support request''']]
* For questions related to most COMSOL data or scripts, contact COMSOL support at:
*: [mailto:support@comsol.com '''support'''@comsol.com]
* '''Exception:''' If your support question might contain data that is subject to '''US Export Control''' regulations, contact instead:
*: [mailto:us-export-control@us.comsol.com '''us-export-control'''@'''us'''.comsol.com]

== The modes to run COMSOL ==
There are several ways of running COMSOL.

=== Running the desktop application ===
To edit models and analyze results, use the COMSOL graphical user interface (GUI). You can run it in several ways, each explained in more detail in the following sections.

# Run on your ''Argonne-provided'' computer, checking out Carbon's license remotely.
#* Requires a full installation of COMSOL on your desktop or laptop computer.
#* You will require access to the COMSOL license server running on Carbon, either through VPN (fairly easy) or SSH tunneling (can be intimidating for first-time users). See below for details.
# Run on Carbon, display on your desktop using the [http://en.wikipedia.org/wiki/X_Window_System X Window system (X11)].
#* Requires an ''X11 Server'' application running on your client machine, and SSH X11 forwarding.
#* The X11 protocol can be slow over LAN.
# Run on Carbon in a [[HPC/Network Access/Virtual Desktop | virtual desktop environment (VNC)]], viewed in a VNC client on your machine.
#* Requires a VNC client on your machine.
#* It can be confusing where VNC and the app runs.
#* The virtual desktop presents a (deliberatly) sparse windowing environment – Copy&Paste can be difficult to use.

=== Running a batch calculation ===
When the computational stage of models takes longer than a few minutes to complete, it will be beneficial to submit save the model file and run it in parallel on Carbon.
See the [[#Batch jobs|section below]].

== Native desktop version ==

To get the fastest response from the COMSOL GUI, run a native COMSOL version on your desktop machine or laptop, called ''client machine'' below.

Enter license details either in the COMSOL Setup configuration dialog.

== Native desktop version — License configuration ==
A client's COMSOL license needs to be configured depending on how the client reaches CNM network resources. Follow the appropriate section:

* [[#License configuration when inside CNM or on CNM VPN|License configuration when '''inside CNM''' or on '''CNM VPN''']]
: Requires that the user's machine is logically within the network of CNM/NST, either by a wired/auth WiFi connection in the building or by VPN under a CNM/NST profile.
* [[#License configuration when outside CNM – requires SSH Tunneling|License configuration when '''outside CNM''' – requires SSH Tunneling]]
: ''Non-NST staff'' usually have a VPN profile that places the machine logically [[#License configuration when outside CNM – requires SSH Tunneling|outside of NST's network.]]

== Use with X11 ==
For casual use, and with [[HPC/Network Access/SSH Tunnel Setup on Linux and MacOS | X11 tunneling to Carbon]] set up, simply add to your <code>~/.bashrc</code> file on Carbon:
module load comsol

Then, from the command line on a Carbon login node, start the COMSOL GUI:
comsol &

License configuration is automatic on Carbon.

Comsol will use a number of cores to run calculations multithreaded by default. (On Carbon, COMSOL_NUM_THREADS is set.)

'''Quit the application''' as soon as you are done. Only one user can use Comsol at a time.

== Use with VNC ==
See general [[HPC/Network Access/Virtual Desktop | instructions on using VNC]], and the COMSOL example therein.

== License configuration when inside CNM or on CNM VPN ==
=== Ensure correct DNS search domains ===
* Inspect the client machine's network configuration.
* Ensure that '''two''' entries are listed as DNS Search Domains for the current network profile:
nst.anl.gov
cnm.anl.gov
: (The order of entries does not matter for the purposes considered here.)
* If the client machine is ''wired to CNM networks'', the domains are normally provided correctly by DHCP.
* If the client machine is ''connected over VPN'', you may need edit the respective network profile.
:* On a Mac, find the VPN configuration details under System Preferences, Network, Advanced… . You may need to unlock the Network panel first before you can make changes under "Advanced…". It appears that those changes are being saved and will be activated automatically the next time that this VPN session gets started.
: [[Image:VNC_add_cnm_to_DNS_1.png|400px]] [[Image:VNC_add_cnm_to_DNS_2.png|400px]]
:* On Windows or Linux machines, take ''analogous'' steps, using the platform's own configuration tools.

* Exclude license manager components from client desktop installation. (Those are all handled server-side.)

=== License format ===
* Floating Network License (FNL), "Three-server redundancy"

=== License servers ===
* On client machines, while running the COMSOL installer or license configuration, specify license servers and ports as follows:
1718 clicense1
1718 clicense2
1718 clicense3
: '''Note:''' Use ''short host names only''. (Technical Reason: The license must be usable both on Carbon's internal network as well as on divisional networks).
1718 clicense1<strike>.cnm.anl.gov</strike>
1718 clicense2<strike>.cnm.anl.gov</strike>
1718 clicense3<strike>.cnm.anl.gov</strike>

'''Advanced:''' A client machine's COMSOL license file <code><COMSOLDIR>/Multiphysics/license/license.dat</code> should look as follows:
<source lang="make">
SERVER clicense1 ANY 1718
SERVER clicense2 ANY 1718
SERVER clicense3 ANY 1718
USE_SERVER
</source>

<code><COMSOLDIR></code> is located by default at:
* '''Windows''': C:\Program files\COMSOL\COMSOL''nn''
* '''Linux''': /usr/local/comsol''nn''
* '''Mac''': /Applications/COMSOL''nn''
("''nn''" is the application's abbreviated version number.)

=== Network Troubleshooting ===
When using the license access model descibed here (as opposed to tunneled over SSH), ensure the following:
* Verify that short host names can be resolved to IP addresses. Either one of the following commands should produce sensible output:
host clicense1
: or:
ping -c 3 clicense1
* Verify that ''both license ports on the same server'' can be reached from your client machine:
telnet '''clicense1''' 1718
telnet '''clicense1''' 1719
: or:
telnet '''clicense2''' 1718
telnet '''clicense2''' 1719
: A ''successful'' connection looks like:
Trying 146.139.x.y...
Connected to clicense1.cnm.anl.gov.
Escape character is '^]'.
:: Do not enter any data. Instead, close each connection by hitting <code>Ctrl</code><code>]</code> (closing square bracket), followed by <code>Ctrl</code><code>D</code>.
: A ''failed'' connection gives one of these errors:
telnet: connect to address 146.139.x.y: Operation timed out
: or:
telnet: connect to address 146.139.x.y: Connection refused
:: In these cases, ask that your IP firewall configuration be reviewed and changed.

== License configuration when outside CNM – requires SSH Tunneling ==
This section documents how to run COMSOL from a network location outside CNM.

As a one-time preparation step, two items must be configured:
* The license host that COMSOL is to contact, and
* An SSH connection that will allow your machine to access the COMSOL license server inside CNM's network.

To then ''run'' the COMSOL application, the configured SSH connection must be up and kept running for the entire time that you run the COMSOL. When you exit COMSOL and wish to start it again, you can keep the SSH connection running, it will be re-used for the next COMSOL run.

=== Configure COMSOL ===
The following needs to be done just once, typically during COMSOL installation:

In the COMSOL Installer, or in COMSOL Preferences, enter:
* '''License format:''' Floating Network License (FNL), "Port number and Hostname"
:: Do ''not use'' "Three-server redundancy", since it cannot (easily) be accessed over SSH. Configure with just one server, and in the rare situation of a server failover, temporarily tweak your SSH configuration to contact the alternate server (see below).
* '''Server:''' <code>127.0.0.1</code>
:: Do not use "localhost", as COMSOL interprets this incorrectly to mean that your machine ''is'' a license server.

* '''Port:''' <code>1718</code>


=== Establish connection to CNM license server ===
This step needs to be done prior to running COMSOL in your desktop session:
* Connect to <code>mega</code> with the following command in a terminal window:
ssh -L 1718:'''clicense1''':1718 -L 1719:'''clicense1''':1719 mega.cnm.anl.gov
* You can leave this connection running in your terminal window across several invocations of COMSOL.

=== Run COMSOL ===
# Ensure your connection to "mega" is still running. If not, restart it.
# Start the COMSOL application.

=== Troubleshooting ===
* If you get an error that the number of licenses is "exhausted", that is confirmation that you correctly reached the license server, though other users are using licenses.
* If the license access attempt takes a long time or hangs, the service might temporarily run on a secondary server. In this case, stop the mega session and restart it, connecting to the alternate license server:
ssh -L 1718:clicense'''2''':1718 -L 1719:clicense'''2''':1719 mega.cnm.anl.gov

== General License Troubleshooting ==
The following are typical error messages.
=== Could not obtain license for COMSOL Multiphysics GUI ===
* Error details:
License error: -4.
'''Licensed number of users already reached.'''
Feature: COMSOLGUI
License path: /Users/stern/Applications/COMSOL52a/license/license.dat:
FlexNet Licensing error:-4,132
* Reason: The connection to the license server succeeded, but the license token(s) you requested are in use.
* Solution: Wait a bit, or [http://carbonmgmt/ganglia/lic.php identify who's using COMSOL] and politely ask the user when they might be finished with their work.

=== License server machine is down or not responding ===
* Error details:
License error: -96.
See the system administrator about starting the license server system, or
make sure you're referring to the right host (see LM_LICENSE_FILE).
Feature: SERIAL
Hostname: clicense3
License path: /Users/stern/Applications/COMSOL52a/license/license.dat:
FlexNet Licensing error:-96,7
* Reason: The license servers cannot be reached at all. The <code>ping</code> or <code>host</code> tests mentioned above fail.
* Solution: Review the client license file and DNS lookup domains.

=== License file does not support this version ===
* Error details:
Feature: SERIAL
'''Application version > License version: 5.21 > 3.5'''
License path: /Users/stern/Applications/COMSOL52a/license/license.dat:
FlexNet Licensing error:-21,126
* Reason: Only the first of two steps for connecting to the license servers succeeds, which may happen when the redundant license service runs on the alternate server instead of the one you're connected to.
* Solution: When using SSH, try <code>clicense2</code> instead of <code>clicense1</code>, or vice versa. Revisit your <code>~/.ssh/config</code> file or use the alternate command line as shown in the SSH section above.

== Multiple CPU cores ==
Ways to start multi-core (from http://www.comsol.com/support/knowledgebase/1096/)
comsol -np 8

export COMSOL_NUM_THREADS=8 # automatically provided on ''Carbon''
comsol
* COMSOL does not benefit from hyperthreading (oversubscribing the physical core count).

== Batch jobs ==

* http://www.comsol.com/support/knowledgebase/1001/ .
* [[Media:HPC 2012-05-16 Comsol parallel usage.pdf]] (Modified from http://www.comsol.com/support/knowledgebase/1001/files/4027/cluster_install_linux_42a.pptx – Appendix slide, PBS)
* '''Note:''' MPD is not supported on Carbon (I patched the main comsol script), and no longer necessary at all in Comsol-4.3.

=== Sample Job file ===
See:
$COMSOL_HOME/sample.job

== Important COMSOL Support Articles ==
* Toplevel suport page: https://www.comsol.com/support

=== KB Articles ===
* [https://www.comsol.com/support/knowledgebase/1001 Running COMSOL® in parallel on clusters]
* [https://www.comsol.com/support/knowledgebase/1250 Running parametric sweeps, batch sweeps, and cluster sweeps from the command line]
* [https://www.comsol.com/support/knowledgebase/1255 Reducing the amount of solution data stored in a model]
* [https://www.comsol.com/support/knowledgebase/830 Handling large models]

=== Blog-style articles ===
* [https://www.comsol.com/blogs/the-power-of-the-batch-sweep/ The Power of the Batch Sweep]
* [https://www.comsol.com/blogs/how-to-run-simulations-in-batch-mode-from-the-command-line/ How to Run Simulations in Batch Mode from the Command Line]

HPC/Applications/comsol

2026-03-04T18:27:27Z

Stern: /* Native desktop version — License configuration */

== Introduction ==
The [http://www.comsol.com/products/multiphysics/ '''COMSOL Multiphysics'''] application suite
is available for Mac, Windows, and Linux.

== Access ==
[[Image:HPC Comsol licensed modules.png|right|450px|thumb|COMSOL features licensed at CNM as of 2024-10.]]
=== COMSOL Multiphysics ===
Access to '''COMSOL Multiphysics''', the main application with both a graphical user interface and batch computing capabilities, will be granted '''''upon request''''' .

You must be:
* An '''''Argonne employee'''''  in one of the following roles:
** '''''NST staff member'''''  (including PostDocs, joint appointments, and students at NST).
** '''''CNM Facility User'''''  (including ''non-NST Argonne employees''), with an active proposal that has COMSOL in scope. Please see [[HPC/FAQ#Access_and_proposals]].

The set of available COMSOL modules that we have licensed is shown at right.

* The native desktop version can be installed only on ''Argonne-owned'' computers (laptop or desktop). Please contact [mailto:help@anl.gov?Subject=COMSOL%20license%20access%20request CNM support]. Make sure to review [http://www.comsol.com/support/updates/ Product Updates].
* To run COMSOL on our HPC system ''Carbon'', as GUI or as batch computing jobs, your proposal must have ''Carbon'' access in scope.

=== COMSOL Server ===
''CNM Facility Users'' who are not Argonne employees are not eligible to access
the full '''COMSOL Multiphysics''' product, in compliance with its licensing terms and conditions.
However, the [https://www.comsol.com/comsol-server '''COMSOL Server'''] product offers an alternative
albeit more narrow means of access.

All users, including '''''non-Argonne users''''' may be able to run, over a web interface in their browser,
pre-parameterized "fixed-model" simulations that were made with the '''COMSOL Application Builder''' (requires Windows).

To inquire about using COMSOL server applications,
please contact us.

We will review possible purchase paths for COMSOL Server, and the ''considerable effort'' likely required by users and the CNM to develop and deploy a COMSOL application.

== Support ==
* For questions on how to run or start COMSOL:
*: Submit an [[HPC/Support | '''IT support request''']]
* For questions related to most COMSOL data or scripts, contact COMSOL support at:
*: [mailto:support@comsol.com '''support'''@comsol.com]
* '''Exception:''' If your support question might contain data that is subject to '''US Export Control''' regulations, contact instead:
*: [mailto:us-export-control@us.comsol.com '''us-export-control'''@'''us'''.comsol.com]

== The modes to run COMSOL ==
There are several ways of running COMSOL.

=== Running the desktop application ===
To edit models and analyze results, use the COMSOL graphical user interface (GUI). You can run it in several ways, each explained in more detail in the following sections.

# Run on your ''Argonne-provided'' computer, checking out Carbon's license remotely.
#* Requires a full installation of COMSOL on your desktop or laptop computer.
#* You will require access to the COMSOL license server running on Carbon, either through VPN (fairly easy) or SSH tunneling (can be intimidating for first-time users). See below for details.
# Run on Carbon, display on your desktop using the [http://en.wikipedia.org/wiki/X_Window_System X Window system (X11)].
#* Requires an ''X11 Server'' application running on your client machine, and SSH X11 forwarding.
#* The X11 protocol can be slow over LAN.
# Run on Carbon in a [[HPC/Network Access/Virtual Desktop | virtual desktop environment (VNC)]], viewed in a VNC client on your machine.
#* Requires a VNC client on your machine.
#* It can be confusing where VNC and the app runs.
#* The virtual desktop presents a (deliberatly) sparse windowing environment – Copy&Paste can be difficult to use.

=== Running a batch calculation ===
When the computational stage of models takes longer than a few minutes to complete, it will be beneficial to submit save the model file and run it in parallel on Carbon.
See the [[#Batch jobs|section below]].

== Native desktop version ==

To get the fastest response from the COMSOL GUI, run a native COMSOL version on your desktop machine or laptop, called ''client machine'' below.

Enter license details either in the COMSOL Setup configuration dialog.

=== License configuration ===
A client's COMSOL license needs to be configured depending on how the client reaches CNM network resources. Follow the appropriate section:

* [[#License configuration when inside CNM or on CNM VPN|License configuration when '''inside CNM''' or on '''CNM VPN''']]
: Requires that the user's machine is logically within the network of CNM/NST, either by a wired/auth WiFi connection in the building or by VPN under a CNM/NST profile.
* [[#License configuration when outside CNM – requires SSH Tunneling|License configuration when '''outside CNM''' – requires SSH Tunneling]]
: ''Non-NST staff'' usually have a VPN profile that places the machine logically [[#License configuration when outside CNM – requires SSH Tunneling|outside of NST's network.]]

== Use with X11 ==
For casual use, and with [[HPC/Network Access/SSH Tunnel Setup on Linux and MacOS | X11 tunneling to Carbon]] set up, simply add to your <code>~/.bashrc</code> file on Carbon:
module load comsol

Then, from the command line on a Carbon login node, start the COMSOL GUI:
comsol &

License configuration is automatic on Carbon.

Comsol will use a number of cores to run calculations multithreaded by default. (On Carbon, COMSOL_NUM_THREADS is set.)

'''Quit the application''' as soon as you are done. Only one user can use Comsol at a time.

== Use with VNC ==
See general [[HPC/Network Access/Virtual Desktop | instructions on using VNC]], and the COMSOL example therein.

== License configuration when inside CNM or on CNM VPN ==
=== Ensure correct DNS search domains ===
* Inspect the client machine's network configuration.
* Ensure that '''two''' entries are listed as DNS Search Domains for the current network profile:
nst.anl.gov
cnm.anl.gov
: (The order of entries does not matter for the purposes considered here.)
* If the client machine is ''wired to CNM networks'', the domains are normally provided correctly by DHCP.
* If the client machine is ''connected over VPN'', you may need edit the respective network profile.
:* On a Mac, find the VPN configuration details under System Preferences, Network, Advanced… . You may need to unlock the Network panel first before you can make changes under "Advanced…". It appears that those changes are being saved and will be activated automatically the next time that this VPN session gets started.
: [[Image:VNC_add_cnm_to_DNS_1.png|400px]] [[Image:VNC_add_cnm_to_DNS_2.png|400px]]
:* On Windows or Linux machines, take ''analogous'' steps, using the platform's own configuration tools.

* Exclude license manager components from client desktop installation. (Those are all handled server-side.)

=== License format ===
* Floating Network License (FNL), "Three-server redundancy"

=== License servers ===
* On client machines, while running the COMSOL installer or license configuration, specify license servers and ports as follows:
1718 clicense1
1718 clicense2
1718 clicense3
: '''Note:''' Use ''short host names only''. (Technical Reason: The license must be usable both on Carbon's internal network as well as on divisional networks).
1718 clicense1<strike>.cnm.anl.gov</strike>
1718 clicense2<strike>.cnm.anl.gov</strike>
1718 clicense3<strike>.cnm.anl.gov</strike>

'''Advanced:''' A client machine's COMSOL license file <code><COMSOLDIR>/Multiphysics/license/license.dat</code> should look as follows:
<source lang="make">
SERVER clicense1 ANY 1718
SERVER clicense2 ANY 1718
SERVER clicense3 ANY 1718
USE_SERVER
</source>

<code><COMSOLDIR></code> is located by default at:
* '''Windows''': C:\Program files\COMSOL\COMSOL''nn''
* '''Linux''': /usr/local/comsol''nn''
* '''Mac''': /Applications/COMSOL''nn''
("''nn''" is the application's abbreviated version number.)

=== Network Troubleshooting ===
When using the license access model descibed here (as opposed to tunneled over SSH), ensure the following:
* Verify that short host names can be resolved to IP addresses. Either one of the following commands should produce sensible output:
host clicense1
: or:
ping -c 3 clicense1
* Verify that ''both license ports on the same server'' can be reached from your client machine:
telnet '''clicense1''' 1718
telnet '''clicense1''' 1719
: or:
telnet '''clicense2''' 1718
telnet '''clicense2''' 1719
: A ''successful'' connection looks like:
Trying 146.139.x.y...
Connected to clicense1.cnm.anl.gov.
Escape character is '^]'.
:: Do not enter any data. Instead, close each connection by hitting <code>Ctrl</code><code>]</code> (closing square bracket), followed by <code>Ctrl</code><code>D</code>.
: A ''failed'' connection gives one of these errors:
telnet: connect to address 146.139.x.y: Operation timed out
: or:
telnet: connect to address 146.139.x.y: Connection refused
:: In these cases, ask that your IP firewall configuration be reviewed and changed.

== License configuration when outside CNM – requires SSH Tunneling ==
This section documents how to run COMSOL from a network location outside CNM.

As a one-time preparation step, two items must be configured:
* The license host that COMSOL is to contact, and
* An SSH connection that will allow your machine to access the COMSOL license server inside CNM's network.

To then ''run'' the COMSOL application, the configured SSH connection must be up and kept running for the entire time that you run the COMSOL. When you exit COMSOL and wish to start it again, you can keep the SSH connection running, it will be re-used for the next COMSOL run.

=== Configure COMSOL ===
The following needs to be done just once, typically during COMSOL installation:

In the COMSOL Installer, or in COMSOL Preferences, enter:
* '''License format:''' Floating Network License (FNL), "Port number and Hostname"
:: Do ''not use'' "Three-server redundancy", since it cannot (easily) be accessed over SSH. Configure with just one server, and in the rare situation of a server failover, temporarily tweak your SSH configuration to contact the alternate server (see below).
* '''Server:''' <code>127.0.0.1</code>
:: Do not use "localhost", as COMSOL interprets this incorrectly to mean that your machine ''is'' a license server.

* '''Port:''' <code>1718</code>


=== Establish connection to CNM license server ===
This step needs to be done prior to running COMSOL in your desktop session:
* Connect to <code>mega</code> with the following command in a terminal window:
ssh -L 1718:'''clicense1''':1718 -L 1719:'''clicense1''':1719 mega.cnm.anl.gov
* You can leave this connection running in your terminal window across several invocations of COMSOL.

=== Run COMSOL ===
# Ensure your connection to "mega" is still running. If not, restart it.
# Start the COMSOL application.

=== Troubleshooting ===
* If you get an error that the number of licenses is "exhausted", that is confirmation that you correctly reached the license server, though other users are using licenses.
* If the license access attempt takes a long time or hangs, the service might temporarily run on a secondary server. In this case, stop the mega session and restart it, connecting to the alternate license server:
ssh -L 1718:clicense'''2''':1718 -L 1719:clicense'''2''':1719 mega.cnm.anl.gov

== General License Troubleshooting ==
The following are typical error messages.
=== Could not obtain license for COMSOL Multiphysics GUI ===
* Error details:
License error: -4.
'''Licensed number of users already reached.'''
Feature: COMSOLGUI
License path: /Users/stern/Applications/COMSOL52a/license/license.dat:
FlexNet Licensing error:-4,132
* Reason: The connection to the license server succeeded, but the license token(s) you requested are in use.
* Solution: Wait a bit, or [http://carbonmgmt/ganglia/lic.php identify who's using COMSOL] and politely ask the user when they might be finished with their work.

=== License server machine is down or not responding ===
* Error details:
License error: -96.
See the system administrator about starting the license server system, or
make sure you're referring to the right host (see LM_LICENSE_FILE).
Feature: SERIAL
Hostname: clicense3
License path: /Users/stern/Applications/COMSOL52a/license/license.dat:
FlexNet Licensing error:-96,7
* Reason: The license servers cannot be reached at all. The <code>ping</code> or <code>host</code> tests mentioned above fail.
* Solution: Review the client license file and DNS lookup domains.

=== License file does not support this version ===
* Error details:
Feature: SERIAL
'''Application version > License version: 5.21 > 3.5'''
License path: /Users/stern/Applications/COMSOL52a/license/license.dat:
FlexNet Licensing error:-21,126
* Reason: Only the first of two steps for connecting to the license servers succeeds, which may happen when the redundant license service runs on the alternate server instead of the one you're connected to.
* Solution: When using SSH, try <code>clicense2</code> instead of <code>clicense1</code>, or vice versa. Revisit your <code>~/.ssh/config</code> file or use the alternate command line as shown in the SSH section above.

== Multiple CPU cores ==
Ways to start multi-core (from http://www.comsol.com/support/knowledgebase/1096/)
comsol -np 8

export COMSOL_NUM_THREADS=8 # automatically provided on ''Carbon''
comsol
* COMSOL does not benefit from hyperthreading (oversubscribing the physical core count).

== Batch jobs ==

* http://www.comsol.com/support/knowledgebase/1001/ .
* [[Media:HPC 2012-05-16 Comsol parallel usage.pdf]] (Modified from http://www.comsol.com/support/knowledgebase/1001/files/4027/cluster_install_linux_42a.pptx – Appendix slide, PBS)
* '''Note:''' MPD is not supported on Carbon (I patched the main comsol script), and no longer necessary at all in Comsol-4.3.

=== Sample Job file ===
See:
$COMSOL_HOME/sample.job

== Important COMSOL Support Articles ==
* Toplevel suport page: https://www.comsol.com/support

=== KB Articles ===
* [https://www.comsol.com/support/knowledgebase/1001 Running COMSOL® in parallel on clusters]
* [https://www.comsol.com/support/knowledgebase/1250 Running parametric sweeps, batch sweeps, and cluster sweeps from the command line]
* [https://www.comsol.com/support/knowledgebase/1255 Reducing the amount of solution data stored in a model]
* [https://www.comsol.com/support/knowledgebase/830 Handling large models]

=== Blog-style articles ===
* [https://www.comsol.com/blogs/the-power-of-the-batch-sweep/ The Power of the Batch Sweep]
* [https://www.comsol.com/blogs/how-to-run-simulations-in-batch-mode-from-the-command-line/ How to Run Simulations in Batch Mode from the Command Line]

HPC/Applications/comsol

2026-03-04T18:26:10Z

Stern:

== Introduction ==
The [http://www.comsol.com/products/multiphysics/ '''COMSOL Multiphysics'''] application suite
is available for Mac, Windows, and Linux.

== Access ==
[[Image:HPC Comsol licensed modules.png|right|450px|thumb|COMSOL features licensed at CNM as of 2024-10.]]
=== COMSOL Multiphysics ===
Access to '''COMSOL Multiphysics''', the main application with both a graphical user interface and batch computing capabilities, will be granted '''''upon request''''' .

You must be:
* An '''''Argonne employee'''''  in one of the following roles:
** '''''NST staff member'''''  (including PostDocs, joint appointments, and students at NST).
** '''''CNM Facility User'''''  (including ''non-NST Argonne employees''), with an active proposal that has COMSOL in scope. Please see [[HPC/FAQ#Access_and_proposals]].

The set of available COMSOL modules that we have licensed is shown at right.

* The native desktop version can be installed only on ''Argonne-owned'' computers (laptop or desktop). Please contact [mailto:help@anl.gov?Subject=COMSOL%20license%20access%20request CNM support]. Make sure to review [http://www.comsol.com/support/updates/ Product Updates].
* To run COMSOL on our HPC system ''Carbon'', as GUI or as batch computing jobs, your proposal must have ''Carbon'' access in scope.

=== COMSOL Server ===
''CNM Facility Users'' who are not Argonne employees are not eligible to access
the full '''COMSOL Multiphysics''' product, in compliance with its licensing terms and conditions.
However, the [https://www.comsol.com/comsol-server '''COMSOL Server'''] product offers an alternative
albeit more narrow means of access.

All users, including '''''non-Argonne users''''' may be able to run, over a web interface in their browser,
pre-parameterized "fixed-model" simulations that were made with the '''COMSOL Application Builder''' (requires Windows).

To inquire about using COMSOL server applications,
please contact us.

We will review possible purchase paths for COMSOL Server, and the ''considerable effort'' likely required by users and the CNM to develop and deploy a COMSOL application.

== Support ==
* For questions on how to run or start COMSOL:
*: Submit an [[HPC/Support | '''IT support request''']]
* For questions related to most COMSOL data or scripts, contact COMSOL support at:
*: [mailto:support@comsol.com '''support'''@comsol.com]
* '''Exception:''' If your support question might contain data that is subject to '''US Export Control''' regulations, contact instead:
*: [mailto:us-export-control@us.comsol.com '''us-export-control'''@'''us'''.comsol.com]

== The modes to run COMSOL ==
There are several ways of running COMSOL.

=== Running the desktop application ===
To edit models and analyze results, use the COMSOL graphical user interface (GUI). You can run it in several ways, each explained in more detail in the following sections.

# Run on your ''Argonne-provided'' computer, checking out Carbon's license remotely.
#* Requires a full installation of COMSOL on your desktop or laptop computer.
#* You will require access to the COMSOL license server running on Carbon, either through VPN (fairly easy) or SSH tunneling (can be intimidating for first-time users). See below for details.
# Run on Carbon, display on your desktop using the [http://en.wikipedia.org/wiki/X_Window_System X Window system (X11)].
#* Requires an ''X11 Server'' application running on your client machine, and SSH X11 forwarding.
#* The X11 protocol can be slow over LAN.
# Run on Carbon in a [[HPC/Network Access/Virtual Desktop | virtual desktop environment (VNC)]], viewed in a VNC client on your machine.
#* Requires a VNC client on your machine.
#* It can be confusing where VNC and the app runs.
#* The virtual desktop presents a (deliberatly) sparse windowing environment – Copy&Paste can be difficult to use.

=== Running a batch calculation ===
When the computational stage of models takes longer than a few minutes to complete, it will be beneficial to submit save the model file and run it in parallel on Carbon.
See the [[#Batch jobs|section below]].

== Native desktop version ==

To get the fastest response from the COMSOL GUI, run a native COMSOL version on your desktop machine or laptop, called ''client machine'' below.

Enter license details either in the COMSOL Setup configuration dialog.

== Native desktop version — License configuration ==
A client's COMSOL license needs to be configured depending on how the client reaches CNM network resources. Follow the appropriate section:

* [[#License configuration when inside CNM or on CNM VPN|License configuration when '''inside CNM''' or on '''CNM VPN''']]
: Requires that the user's machine is logically within the network of CNM/NST, either by a wired/auth WiFi connection in the building or by VPN under a CNM/NST profile.
* [[#License configuration when outside CNM – requires SSH Tunneling|License configuration when '''outside CNM''' – requires SSH Tunneling]]
: ''Non-NST staff'' usually have a VPN profile that places the machine logically [[#License configuration when outside CNM – requires SSH Tunneling|outside of NST's network.]]

== Use with X11 ==
For casual use, and with [[HPC/Network Access/SSH Tunnel Setup on Linux and MacOS | X11 tunneling to Carbon]] set up, simply add to your <code>~/.bashrc</code> file on Carbon:
module load comsol

Then, from the command line on a Carbon login node, start the COMSOL GUI:
comsol &

License configuration is automatic on Carbon.

Comsol will use a number of cores to run calculations multithreaded by default. (On Carbon, COMSOL_NUM_THREADS is set.)

'''Quit the application''' as soon as you are done. Only one user can use Comsol at a time.

== Use with VNC ==
See general [[HPC/Network Access/Virtual Desktop | instructions on using VNC]], and the COMSOL example therein.

== License configuration when inside CNM or on CNM VPN ==
=== Ensure correct DNS search domains ===
* Inspect the client machine's network configuration.
* Ensure that '''two''' entries are listed as DNS Search Domains for the current network profile:
nst.anl.gov
cnm.anl.gov
: (The order of entries does not matter for the purposes considered here.)
* If the client machine is ''wired to CNM networks'', the domains are normally provided correctly by DHCP.
* If the client machine is ''connected over VPN'', you may need edit the respective network profile.
:* On a Mac, find the VPN configuration details under System Preferences, Network, Advanced… . You may need to unlock the Network panel first before you can make changes under "Advanced…". It appears that those changes are being saved and will be activated automatically the next time that this VPN session gets started.
: [[Image:VNC_add_cnm_to_DNS_1.png|400px]] [[Image:VNC_add_cnm_to_DNS_2.png|400px]]
:* On Windows or Linux machines, take ''analogous'' steps, using the platform's own configuration tools.

* Exclude license manager components from client desktop installation. (Those are all handled server-side.)

=== License format ===
* Floating Network License (FNL), "Three-server redundancy"

=== License servers ===
* On client machines, while running the COMSOL installer or license configuration, specify license servers and ports as follows:
1718 clicense1
1718 clicense2
1718 clicense3
: '''Note:''' Use ''short host names only''. (Technical Reason: The license must be usable both on Carbon's internal network as well as on divisional networks).
1718 clicense1<strike>.cnm.anl.gov</strike>
1718 clicense2<strike>.cnm.anl.gov</strike>
1718 clicense3<strike>.cnm.anl.gov</strike>

'''Advanced:''' A client machine's COMSOL license file <code><COMSOLDIR>/Multiphysics/license/license.dat</code> should look as follows:
<source lang="make">
SERVER clicense1 ANY 1718
SERVER clicense2 ANY 1718
SERVER clicense3 ANY 1718
USE_SERVER
</source>

<code><COMSOLDIR></code> is located by default at:
* '''Windows''': C:\Program files\COMSOL\COMSOL''nn''
* '''Linux''': /usr/local/comsol''nn''
* '''Mac''': /Applications/COMSOL''nn''
("''nn''" is the application's abbreviated version number.)

=== Network Troubleshooting ===
When using the license access model descibed here (as opposed to tunneled over SSH), ensure the following:
* Verify that short host names can be resolved to IP addresses. Either one of the following commands should produce sensible output:
host clicense1
: or:
ping -c 3 clicense1
* Verify that ''both license ports on the same server'' can be reached from your client machine:
telnet '''clicense1''' 1718
telnet '''clicense1''' 1719
: or:
telnet '''clicense2''' 1718
telnet '''clicense2''' 1719
: A ''successful'' connection looks like:
Trying 146.139.x.y...
Connected to clicense1.cnm.anl.gov.
Escape character is '^]'.
:: Do not enter any data. Instead, close each connection by hitting <code>Ctrl</code><code>]</code> (closing square bracket), followed by <code>Ctrl</code><code>D</code>.
: A ''failed'' connection gives one of these errors:
telnet: connect to address 146.139.x.y: Operation timed out
: or:
telnet: connect to address 146.139.x.y: Connection refused
:: In these cases, ask that your IP firewall configuration be reviewed and changed.

== License configuration when outside CNM – requires SSH Tunneling ==
This section documents how to run COMSOL from a network location outside CNM.

As a one-time preparation step, two items must be configured:
* The license host that COMSOL is to contact, and
* An SSH connection that will allow your machine to access the COMSOL license server inside CNM's network.

To then ''run'' the COMSOL application, the configured SSH connection must be up and kept running for the entire time that you run the COMSOL. When you exit COMSOL and wish to start it again, you can keep the SSH connection running, it will be re-used for the next COMSOL run.

=== Configure COMSOL ===
The following needs to be done just once, typically during COMSOL installation:

In the COMSOL Installer, or in COMSOL Preferences, enter:
* '''License format:''' Floating Network License (FNL), "Port number and Hostname"
:: Do ''not use'' "Three-server redundancy", since it cannot (easily) be accessed over SSH. Configure with just one server, and in the rare situation of a server failover, temporarily tweak your SSH configuration to contact the alternate server (see below).
* '''Server:''' <code>127.0.0.1</code>
:: Do not use "localhost", as COMSOL interprets this incorrectly to mean that your machine ''is'' a license server.

* '''Port:''' <code>1718</code>


=== Establish connection to CNM license server ===
This step needs to be done prior to running COMSOL in your desktop session:
* Connect to <code>mega</code> with the following command in a terminal window:
ssh -L 1718:'''clicense1''':1718 -L 1719:'''clicense1''':1719 mega.cnm.anl.gov
* You can leave this connection running in your terminal window across several invocations of COMSOL.

=== Run COMSOL ===
# Ensure your connection to "mega" is still running. If not, restart it.
# Start the COMSOL application.

=== Troubleshooting ===
* If you get an error that the number of licenses is "exhausted", that is confirmation that you correctly reached the license server, though other users are using licenses.
* If the license access attempt takes a long time or hangs, the service might temporarily run on a secondary server. In this case, stop the mega session and restart it, connecting to the alternate license server:
ssh -L 1718:clicense'''2''':1718 -L 1719:clicense'''2''':1719 mega.cnm.anl.gov

== General License Troubleshooting ==
The following are typical error messages.
=== Could not obtain license for COMSOL Multiphysics GUI ===
* Error details:
License error: -4.
'''Licensed number of users already reached.'''
Feature: COMSOLGUI
License path: /Users/stern/Applications/COMSOL52a/license/license.dat:
FlexNet Licensing error:-4,132
* Reason: The connection to the license server succeeded, but the license token(s) you requested are in use.
* Solution: Wait a bit, or [http://carbonmgmt/ganglia/lic.php identify who's using COMSOL] and politely ask the user when they might be finished with their work.

=== License server machine is down or not responding ===
* Error details:
License error: -96.
See the system administrator about starting the license server system, or
make sure you're referring to the right host (see LM_LICENSE_FILE).
Feature: SERIAL
Hostname: clicense3
License path: /Users/stern/Applications/COMSOL52a/license/license.dat:
FlexNet Licensing error:-96,7
* Reason: The license servers cannot be reached at all. The <code>ping</code> or <code>host</code> tests mentioned above fail.
* Solution: Review the client license file and DNS lookup domains.

=== License file does not support this version ===
* Error details:
Feature: SERIAL
'''Application version > License version: 5.21 > 3.5'''
License path: /Users/stern/Applications/COMSOL52a/license/license.dat:
FlexNet Licensing error:-21,126
* Reason: Only the first of two steps for connecting to the license servers succeeds, which may happen when the redundant license service runs on the alternate server instead of the one you're connected to.
* Solution: When using SSH, try <code>clicense2</code> instead of <code>clicense1</code>, or vice versa. Revisit your <code>~/.ssh/config</code> file or use the alternate command line as shown in the SSH section above.

== Multiple CPU cores ==
Ways to start multi-core (from http://www.comsol.com/support/knowledgebase/1096/)
comsol -np 8

export COMSOL_NUM_THREADS=8 # automatically provided on ''Carbon''
comsol
* COMSOL does not benefit from hyperthreading (oversubscribing the physical core count).

== Batch jobs ==

* http://www.comsol.com/support/knowledgebase/1001/ .
* [[Media:HPC 2012-05-16 Comsol parallel usage.pdf]] (Modified from http://www.comsol.com/support/knowledgebase/1001/files/4027/cluster_install_linux_42a.pptx – Appendix slide, PBS)
* '''Note:''' MPD is not supported on Carbon (I patched the main comsol script), and no longer necessary at all in Comsol-4.3.

=== Sample Job file ===
See:
$COMSOL_HOME/sample.job

== Important COMSOL Support Articles ==
* Toplevel suport page: https://www.comsol.com/support

=== KB Articles ===
* [https://www.comsol.com/support/knowledgebase/1001 Running COMSOL® in parallel on clusters]
* [https://www.comsol.com/support/knowledgebase/1250 Running parametric sweeps, batch sweeps, and cluster sweeps from the command line]
* [https://www.comsol.com/support/knowledgebase/1255 Reducing the amount of solution data stored in a model]
* [https://www.comsol.com/support/knowledgebase/830 Handling large models]

=== Blog-style articles ===
* [https://www.comsol.com/blogs/the-power-of-the-batch-sweep/ The Power of the Batch Sweep]
* [https://www.comsol.com/blogs/how-to-run-simulations-in-batch-mode-from-the-command-line/ How to Run Simulations in Batch Mode from the Command Line]

HPC/Applications/comsol

2026-03-04T18:24:55Z

Stern: /* Running the desktop application */

== Introduction ==
The [http://www.comsol.com/products/multiphysics/ '''COMSOL Multiphysics'''] application suite
is available for Mac, Windows, and Linux.

== Access ==
[[Image:HPC Comsol licensed modules.png|right|450px|thumb|COMSOL features licensed at CNM as of 2024-10.]]
=== COMSOL Multiphysics ===
Access to '''COMSOL Multiphysics''', the main application with both a graphical user interface and batch computing capabilities, will be granted '''''upon request''''' .

You must be:
* An '''''Argonne employee'''''  in one of the following roles:
** '''''NST staff member'''''  (including PostDocs, joint appointments, and students at NST).
** '''''CNM Facility User'''''  (including ''non-NST Argonne employees''), with an active proposal that has COMSOL in scope. Please see [[HPC/FAQ#Access_and_proposals]].

The set of available COMSOL modules that we have licensed is shown at right.

* The native desktop version can be installed only on ''Argonne-owned'' computers (laptop or desktop). Please contact [mailto:help@anl.gov?Subject=COMSOL%20license%20access%20request CNM support]. Make sure to review [http://www.comsol.com/support/updates/ Product Updates].
* To run COMSOL on our HPC system ''Carbon'', as GUI or as batch computing jobs, your proposal must have ''Carbon'' access in scope.

=== COMSOL Server ===
''CNM Facility Users'' who are not Argonne employees are not eligible to access
the full '''COMSOL Multiphysics''' product, in compliance with its licensing terms and conditions.
However, the [https://www.comsol.com/comsol-server '''COMSOL Server'''] product offers an alternative
albeit more narrow means of access.

All users, including '''''non-Argonne users''''' may be able to run, over a web interface in their browser,
pre-parameterized "fixed-model" simulations that were made with the '''COMSOL Application Builder''' (requires Windows).

To inquire about using COMSOL server applications,
please contact us.

We will review possible purchase paths for COMSOL Server, and the ''considerable effort'' likely required by users and the CNM to develop and deploy a COMSOL application.

== Support ==
* For questions on how to run or start COMSOL:
*: Submit an [[HPC/Support | '''IT support request''']]
* For questions related to most COMSOL data or scripts, contact COMSOL support at:
*: [mailto:support@comsol.com '''support'''@comsol.com]
* '''Exception:''' If your support question might contain data that is subject to '''US Export Control''' regulations, contact instead:
*: [mailto:us-export-control@us.comsol.com '''us-export-control'''@'''us'''.comsol.com]

== The modes to run COMSOL ==
There are several ways of running COMSOL.

=== Running the desktop application ===
To edit models and analyze results, use the COMSOL graphical user interface (GUI). You can run it in several ways, each explained in more detail in the following sections.

# Run on your ''Argonne-provided'' computer, checking out Carbon's license remotely.
#* Requires a full installation of COMSOL on your desktop or laptop computer.
#* You will require access to the COMSOL license server running on Carbon, either through VPN (fairly easy) or SSH tunneling (can be intimidating for first-time users). See below for details.
# Run on Carbon, display on your desktop using the [http://en.wikipedia.org/wiki/X_Window_System X Window system (X11)].
#* Requires an ''X11 Server'' application running on your client machine, and SSH X11 forwarding.
#* The X11 protocol can be slow over LAN.
# Run on Carbon in a [[HPC/Network Access/Virtual Desktop | virtual desktop environment (VNC)]], viewed in a VNC client on your machine.
#* Requires a VNC client on your machine.
#* It can be confusing where VNC and the app runs.
#* The virtual desktop presents a (deliberatly) sparse windowing environment – Copy&Paste can be difficult to use.

=== Running a batch calculation ===
When the computational stage of models takes longer than a few minutes to complete, it will be beneficial to submit save the model file and run it in parallel on Carbon.
See the [[#Batch jobs|section below]].

== Use with X11 ==
For casual use, and with [[HPC/Network Access/SSH Tunnel Setup on Linux and MacOS | X11 tunneling to Carbon]] set up, simply add to your <code>~/.bashrc</code> file on Carbon:
module load comsol

Then, from the command line on a Carbon login node, start the COMSOL GUI:
comsol &

License configuration is automatic on Carbon.

Comsol will use a number of cores to run calculations multithreaded by default. (On Carbon, COMSOL_NUM_THREADS is set.)

'''Quit the application''' as soon as you are done. Only one user can use Comsol at a time.

== Use with VNC ==
See general [[HPC/Network Access/Virtual Desktop | instructions on using VNC]], and the COMSOL example therein.

== Native desktop version ==

To get the fastest response from the COMSOL GUI, run a native COMSOL version on your desktop machine or laptop, called ''client machine'' below.

Enter license details either in the COMSOL Setup configuration dialog.

== Native desktop version — License configuration ==
A client's COMSOL license needs to be configured depending on how the client reaches CNM network resources. Follow the appropriate section:

* [[#License configuration when inside CNM or on CNM VPN|License configuration when '''inside CNM''' or on '''CNM VPN''']]
: Requires that the user's machine is logically within the network of CNM/NST, either by a wired/auth WiFi connection in the building or by VPN under a CNM/NST profile.
* [[#License configuration when outside CNM – requires SSH Tunneling|License configuration when '''outside CNM''' – requires SSH Tunneling]]
: ''Non-NST staff'' usually have a VPN profile that places the machine logically [[#License configuration when outside CNM – requires SSH Tunneling|outside of NST's network.]]

== License configuration when inside CNM or on CNM VPN ==
=== Ensure correct DNS search domains ===
* Inspect the client machine's network configuration.
* Ensure that '''two''' entries are listed as DNS Search Domains for the current network profile:
nst.anl.gov
cnm.anl.gov
: (The order of entries does not matter for the purposes considered here.)
* If the client machine is ''wired to CNM networks'', the domains are normally provided correctly by DHCP.
* If the client machine is ''connected over VPN'', you may need edit the respective network profile.
:* On a Mac, find the VPN configuration details under System Preferences, Network, Advanced… . You may need to unlock the Network panel first before you can make changes under "Advanced…". It appears that those changes are being saved and will be activated automatically the next time that this VPN session gets started.
: [[Image:VNC_add_cnm_to_DNS_1.png|400px]] [[Image:VNC_add_cnm_to_DNS_2.png|400px]]
:* On Windows or Linux machines, take ''analogous'' steps, using the platform's own configuration tools.

* Exclude license manager components from client desktop installation. (Those are all handled server-side.)

=== License format ===
* Floating Network License (FNL), "Three-server redundancy"

=== License servers ===
* On client machines, while running the COMSOL installer or license configuration, specify license servers and ports as follows:
1718 clicense1
1718 clicense2
1718 clicense3
: '''Note:''' Use ''short host names only''. (Technical Reason: The license must be usable both on Carbon's internal network as well as on divisional networks).
1718 clicense1<strike>.cnm.anl.gov</strike>
1718 clicense2<strike>.cnm.anl.gov</strike>
1718 clicense3<strike>.cnm.anl.gov</strike>

'''Advanced:''' A client machine's COMSOL license file <code><COMSOLDIR>/Multiphysics/license/license.dat</code> should look as follows:
<source lang="make">
SERVER clicense1 ANY 1718
SERVER clicense2 ANY 1718
SERVER clicense3 ANY 1718
USE_SERVER
</source>

<code><COMSOLDIR></code> is located by default at:
* '''Windows''': C:\Program files\COMSOL\COMSOL''nn''
* '''Linux''': /usr/local/comsol''nn''
* '''Mac''': /Applications/COMSOL''nn''
("''nn''" is the application's abbreviated version number.)

=== Network Troubleshooting ===
When using the license access model descibed here (as opposed to tunneled over SSH), ensure the following:
* Verify that short host names can be resolved to IP addresses. Either one of the following commands should produce sensible output:
host clicense1
: or:
ping -c 3 clicense1
* Verify that ''both license ports on the same server'' can be reached from your client machine:
telnet '''clicense1''' 1718
telnet '''clicense1''' 1719
: or:
telnet '''clicense2''' 1718
telnet '''clicense2''' 1719
: A ''successful'' connection looks like:
Trying 146.139.x.y...
Connected to clicense1.cnm.anl.gov.
Escape character is '^]'.
:: Do not enter any data. Instead, close each connection by hitting <code>Ctrl</code><code>]</code> (closing square bracket), followed by <code>Ctrl</code><code>D</code>.
: A ''failed'' connection gives one of these errors:
telnet: connect to address 146.139.x.y: Operation timed out
: or:
telnet: connect to address 146.139.x.y: Connection refused
:: In these cases, ask that your IP firewall configuration be reviewed and changed.

== License configuration when outside CNM – requires SSH Tunneling ==
This section documents how to run COMSOL from a network location outside CNM.

As a one-time preparation step, two items must be configured:
* The license host that COMSOL is to contact, and
* An SSH connection that will allow your machine to access the COMSOL license server inside CNM's network.

To then ''run'' the COMSOL application, the configured SSH connection must be up and kept running for the entire time that you run the COMSOL. When you exit COMSOL and wish to start it again, you can keep the SSH connection running, it will be re-used for the next COMSOL run.

=== Configure COMSOL ===
The following needs to be done just once, typically during COMSOL installation:

In the COMSOL Installer, or in COMSOL Preferences, enter:
* '''License format:''' Floating Network License (FNL), "Port number and Hostname"
:: Do ''not use'' "Three-server redundancy", since it cannot (easily) be accessed over SSH. Configure with just one server, and in the rare situation of a server failover, temporarily tweak your SSH configuration to contact the alternate server (see below).
* '''Server:''' <code>127.0.0.1</code>
:: Do not use "localhost", as COMSOL interprets this incorrectly to mean that your machine ''is'' a license server.

* '''Port:''' <code>1718</code>


=== Establish connection to CNM license server ===
This step needs to be done prior to running COMSOL in your desktop session:
* Connect to <code>mega</code> with the following command in a terminal window:
ssh -L 1718:'''clicense1''':1718 -L 1719:'''clicense1''':1719 mega.cnm.anl.gov
* You can leave this connection running in your terminal window across several invocations of COMSOL.

=== Run COMSOL ===
# Ensure your connection to "mega" is still running. If not, restart it.
# Start the COMSOL application.

=== Troubleshooting ===
* If you get an error that the number of licenses is "exhausted", that is confirmation that you correctly reached the license server, though other users are using licenses.
* If the license access attempt takes a long time or hangs, the service might temporarily run on a secondary server. In this case, stop the mega session and restart it, connecting to the alternate license server:
ssh -L 1718:clicense'''2''':1718 -L 1719:clicense'''2''':1719 mega.cnm.anl.gov

== General License Troubleshooting ==
The following are typical error messages.
=== Could not obtain license for COMSOL Multiphysics GUI ===
* Error details:
License error: -4.
'''Licensed number of users already reached.'''
Feature: COMSOLGUI
License path: /Users/stern/Applications/COMSOL52a/license/license.dat:
FlexNet Licensing error:-4,132
* Reason: The connection to the license server succeeded, but the license token(s) you requested are in use.
* Solution: Wait a bit, or [http://carbonmgmt/ganglia/lic.php identify who's using COMSOL] and politely ask the user when they might be finished with their work.

=== License server machine is down or not responding ===
* Error details:
License error: -96.
See the system administrator about starting the license server system, or
make sure you're referring to the right host (see LM_LICENSE_FILE).
Feature: SERIAL
Hostname: clicense3
License path: /Users/stern/Applications/COMSOL52a/license/license.dat:
FlexNet Licensing error:-96,7
* Reason: The license servers cannot be reached at all. The <code>ping</code> or <code>host</code> tests mentioned above fail.
* Solution: Review the client license file and DNS lookup domains.

=== License file does not support this version ===
* Error details:
Feature: SERIAL
'''Application version > License version: 5.21 > 3.5'''
License path: /Users/stern/Applications/COMSOL52a/license/license.dat:
FlexNet Licensing error:-21,126
* Reason: Only the first of two steps for connecting to the license servers succeeds, which may happen when the redundant license service runs on the alternate server instead of the one you're connected to.
* Solution: When using SSH, try <code>clicense2</code> instead of <code>clicense1</code>, or vice versa. Revisit your <code>~/.ssh/config</code> file or use the alternate command line as shown in the SSH section above.

== Multiple CPU cores ==
Ways to start multi-core (from http://www.comsol.com/support/knowledgebase/1096/)
comsol -np 8

export COMSOL_NUM_THREADS=8 # automatically provided on ''Carbon''
comsol
* COMSOL does not benefit from hyperthreading (oversubscribing the physical core count).

== Batch jobs ==

* http://www.comsol.com/support/knowledgebase/1001/ .
* [[Media:HPC 2012-05-16 Comsol parallel usage.pdf]] (Modified from http://www.comsol.com/support/knowledgebase/1001/files/4027/cluster_install_linux_42a.pptx – Appendix slide, PBS)
* '''Note:''' MPD is not supported on Carbon (I patched the main comsol script), and no longer necessary at all in Comsol-4.3.

=== Sample Job file ===
See:
$COMSOL_HOME/sample.job

== Important COMSOL Support Articles ==
* Toplevel suport page: https://www.comsol.com/support

=== KB Articles ===
* [https://www.comsol.com/support/knowledgebase/1001 Running COMSOL® in parallel on clusters]
* [https://www.comsol.com/support/knowledgebase/1250 Running parametric sweeps, batch sweeps, and cluster sweeps from the command line]
* [https://www.comsol.com/support/knowledgebase/1255 Reducing the amount of solution data stored in a model]
* [https://www.comsol.com/support/knowledgebase/830 Handling large models]

=== Blog-style articles ===
* [https://www.comsol.com/blogs/the-power-of-the-batch-sweep/ The Power of the Batch Sweep]
* [https://www.comsol.com/blogs/how-to-run-simulations-in-batch-mode-from-the-command-line/ How to Run Simulations in Batch Mode from the Command Line]

HPC/Applications/comsol

2026-03-04T18:24:43Z

Stern: /* Running the desktop application */

== Introduction ==
The [http://www.comsol.com/products/multiphysics/ '''COMSOL Multiphysics'''] application suite
is available for Mac, Windows, and Linux.

== Access ==
[[Image:HPC Comsol licensed modules.png|right|450px|thumb|COMSOL features licensed at CNM as of 2024-10.]]
=== COMSOL Multiphysics ===
Access to '''COMSOL Multiphysics''', the main application with both a graphical user interface and batch computing capabilities, will be granted '''''upon request''''' .

You must be:
* An '''''Argonne employee'''''  in one of the following roles:
** '''''NST staff member'''''  (including PostDocs, joint appointments, and students at NST).
** '''''CNM Facility User'''''  (including ''non-NST Argonne employees''), with an active proposal that has COMSOL in scope. Please see [[HPC/FAQ#Access_and_proposals]].

The set of available COMSOL modules that we have licensed is shown at right.

* The native desktop version can be installed only on ''Argonne-owned'' computers (laptop or desktop). Please contact [mailto:help@anl.gov?Subject=COMSOL%20license%20access%20request CNM support]. Make sure to review [http://www.comsol.com/support/updates/ Product Updates].
* To run COMSOL on our HPC system ''Carbon'', as GUI or as batch computing jobs, your proposal must have ''Carbon'' access in scope.

=== COMSOL Server ===
''CNM Facility Users'' who are not Argonne employees are not eligible to access
the full '''COMSOL Multiphysics''' product, in compliance with its licensing terms and conditions.
However, the [https://www.comsol.com/comsol-server '''COMSOL Server'''] product offers an alternative
albeit more narrow means of access.

All users, including '''''non-Argonne users''''' may be able to run, over a web interface in their browser,
pre-parameterized "fixed-model" simulations that were made with the '''COMSOL Application Builder''' (requires Windows).

To inquire about using COMSOL server applications,
please contact us.

We will review possible purchase paths for COMSOL Server, and the ''considerable effort'' likely required by users and the CNM to develop and deploy a COMSOL application.

== Support ==
* For questions on how to run or start COMSOL:
*: Submit an [[HPC/Support | '''IT support request''']]
* For questions related to most COMSOL data or scripts, contact COMSOL support at:
*: [mailto:support@comsol.com '''support'''@comsol.com]
* '''Exception:''' If your support question might contain data that is subject to '''US Export Control''' regulations, contact instead:
*: [mailto:us-export-control@us.comsol.com '''us-export-control'''@'''us'''.comsol.com]

== The modes to run COMSOL ==
There are several ways of running COMSOL.

=== Running the desktop application ===
To edit models and analyze results, use the COMSOL graphical user interface (GUI). You can run it in several ways, each explained in more detail in the following sections.

# Run on your ''Argonne-provided'' computer, checking out Carbon's license remotely.
#* Requires a full installation of COMSOL on your desktoo or laptop computer.
#* You will require access to the COMSOL license server running on Carbon, either through VPN (fairly easy) or SSH tunneling (can be intimidating for first-time users). See below for details.
# Run on Carbon, display on your desktop using the [http://en.wikipedia.org/wiki/X_Window_System X Window system (X11)].
#* Requires an ''X11 Server'' application running on your client machine, and SSH X11 forwarding.
#* The X11 protocol can be slow over LAN.
# Run on Carbon in a [[HPC/Network Access/Virtual Desktop | virtual desktop environment (VNC)]], viewed in a VNC client on your machine.
#* Requires a VNC client on your machine.
#* It can be confusing where VNC and the app runs.
#* The virtual desktop presents a (deliberatly) sparse windowing environment – Copy&Paste can be difficult to use.

=== Running a batch calculation ===
When the computational stage of models takes longer than a few minutes to complete, it will be beneficial to submit save the model file and run it in parallel on Carbon.
See the [[#Batch jobs|section below]].

== Use with X11 ==
For casual use, and with [[HPC/Network Access/SSH Tunnel Setup on Linux and MacOS | X11 tunneling to Carbon]] set up, simply add to your <code>~/.bashrc</code> file on Carbon:
module load comsol

Then, from the command line on a Carbon login node, start the COMSOL GUI:
comsol &

License configuration is automatic on Carbon.

Comsol will use a number of cores to run calculations multithreaded by default. (On Carbon, COMSOL_NUM_THREADS is set.)

'''Quit the application''' as soon as you are done. Only one user can use Comsol at a time.

== Use with VNC ==
See general [[HPC/Network Access/Virtual Desktop | instructions on using VNC]], and the COMSOL example therein.

== Native desktop version ==

To get the fastest response from the COMSOL GUI, run a native COMSOL version on your desktop machine or laptop, called ''client machine'' below.

Enter license details either in the COMSOL Setup configuration dialog.

== Native desktop version — License configuration ==
A client's COMSOL license needs to be configured depending on how the client reaches CNM network resources. Follow the appropriate section:

* [[#License configuration when inside CNM or on CNM VPN|License configuration when '''inside CNM''' or on '''CNM VPN''']]
: Requires that the user's machine is logically within the network of CNM/NST, either by a wired/auth WiFi connection in the building or by VPN under a CNM/NST profile.
* [[#License configuration when outside CNM – requires SSH Tunneling|License configuration when '''outside CNM''' – requires SSH Tunneling]]
: ''Non-NST staff'' usually have a VPN profile that places the machine logically [[#License configuration when outside CNM – requires SSH Tunneling|outside of NST's network.]]

== License configuration when inside CNM or on CNM VPN ==
=== Ensure correct DNS search domains ===
* Inspect the client machine's network configuration.
* Ensure that '''two''' entries are listed as DNS Search Domains for the current network profile:
nst.anl.gov
cnm.anl.gov
: (The order of entries does not matter for the purposes considered here.)
* If the client machine is ''wired to CNM networks'', the domains are normally provided correctly by DHCP.
* If the client machine is ''connected over VPN'', you may need edit the respective network profile.
:* On a Mac, find the VPN configuration details under System Preferences, Network, Advanced… . You may need to unlock the Network panel first before you can make changes under "Advanced…". It appears that those changes are being saved and will be activated automatically the next time that this VPN session gets started.
: [[Image:VNC_add_cnm_to_DNS_1.png|400px]] [[Image:VNC_add_cnm_to_DNS_2.png|400px]]
:* On Windows or Linux machines, take ''analogous'' steps, using the platform's own configuration tools.

* Exclude license manager components from client desktop installation. (Those are all handled server-side.)

=== License format ===
* Floating Network License (FNL), "Three-server redundancy"

=== License servers ===
* On client machines, while running the COMSOL installer or license configuration, specify license servers and ports as follows:
1718 clicense1
1718 clicense2
1718 clicense3
: '''Note:''' Use ''short host names only''. (Technical Reason: The license must be usable both on Carbon's internal network as well as on divisional networks).
1718 clicense1<strike>.cnm.anl.gov</strike>
1718 clicense2<strike>.cnm.anl.gov</strike>
1718 clicense3<strike>.cnm.anl.gov</strike>

'''Advanced:''' A client machine's COMSOL license file <code><COMSOLDIR>/Multiphysics/license/license.dat</code> should look as follows:
<source lang="make">
SERVER clicense1 ANY 1718
SERVER clicense2 ANY 1718
SERVER clicense3 ANY 1718
USE_SERVER
</source>

<code><COMSOLDIR></code> is located by default at:
* '''Windows''': C:\Program files\COMSOL\COMSOL''nn''
* '''Linux''': /usr/local/comsol''nn''
* '''Mac''': /Applications/COMSOL''nn''
("''nn''" is the application's abbreviated version number.)

=== Network Troubleshooting ===
When using the license access model descibed here (as opposed to tunneled over SSH), ensure the following:
* Verify that short host names can be resolved to IP addresses. Either one of the following commands should produce sensible output:
host clicense1
: or:
ping -c 3 clicense1
* Verify that ''both license ports on the same server'' can be reached from your client machine:
telnet '''clicense1''' 1718
telnet '''clicense1''' 1719
: or:
telnet '''clicense2''' 1718
telnet '''clicense2''' 1719
: A ''successful'' connection looks like:
Trying 146.139.x.y...
Connected to clicense1.cnm.anl.gov.
Escape character is '^]'.
:: Do not enter any data. Instead, close each connection by hitting <code>Ctrl</code><code>]</code> (closing square bracket), followed by <code>Ctrl</code><code>D</code>.
: A ''failed'' connection gives one of these errors:
telnet: connect to address 146.139.x.y: Operation timed out
: or:
telnet: connect to address 146.139.x.y: Connection refused
:: In these cases, ask that your IP firewall configuration be reviewed and changed.

== License configuration when outside CNM – requires SSH Tunneling ==
This section documents how to run COMSOL from a network location outside CNM.

As a one-time preparation step, two items must be configured:
* The license host that COMSOL is to contact, and
* An SSH connection that will allow your machine to access the COMSOL license server inside CNM's network.

To then ''run'' the COMSOL application, the configured SSH connection must be up and kept running for the entire time that you run the COMSOL. When you exit COMSOL and wish to start it again, you can keep the SSH connection running, it will be re-used for the next COMSOL run.

=== Configure COMSOL ===
The following needs to be done just once, typically during COMSOL installation:

In the COMSOL Installer, or in COMSOL Preferences, enter:
* '''License format:''' Floating Network License (FNL), "Port number and Hostname"
:: Do ''not use'' "Three-server redundancy", since it cannot (easily) be accessed over SSH. Configure with just one server, and in the rare situation of a server failover, temporarily tweak your SSH configuration to contact the alternate server (see below).
* '''Server:''' <code>127.0.0.1</code>
:: Do not use "localhost", as COMSOL interprets this incorrectly to mean that your machine ''is'' a license server.

* '''Port:''' <code>1718</code>


=== Establish connection to CNM license server ===
This step needs to be done prior to running COMSOL in your desktop session:
* Connect to <code>mega</code> with the following command in a terminal window:
ssh -L 1718:'''clicense1''':1718 -L 1719:'''clicense1''':1719 mega.cnm.anl.gov
* You can leave this connection running in your terminal window across several invocations of COMSOL.

=== Run COMSOL ===
# Ensure your connection to "mega" is still running. If not, restart it.
# Start the COMSOL application.

=== Troubleshooting ===
* If you get an error that the number of licenses is "exhausted", that is confirmation that you correctly reached the license server, though other users are using licenses.
* If the license access attempt takes a long time or hangs, the service might temporarily run on a secondary server. In this case, stop the mega session and restart it, connecting to the alternate license server:
ssh -L 1718:clicense'''2''':1718 -L 1719:clicense'''2''':1719 mega.cnm.anl.gov

== General License Troubleshooting ==
The following are typical error messages.
=== Could not obtain license for COMSOL Multiphysics GUI ===
* Error details:
License error: -4.
'''Licensed number of users already reached.'''
Feature: COMSOLGUI
License path: /Users/stern/Applications/COMSOL52a/license/license.dat:
FlexNet Licensing error:-4,132
* Reason: The connection to the license server succeeded, but the license token(s) you requested are in use.
* Solution: Wait a bit, or [http://carbonmgmt/ganglia/lic.php identify who's using COMSOL] and politely ask the user when they might be finished with their work.

=== License server machine is down or not responding ===
* Error details:
License error: -96.
See the system administrator about starting the license server system, or
make sure you're referring to the right host (see LM_LICENSE_FILE).
Feature: SERIAL
Hostname: clicense3
License path: /Users/stern/Applications/COMSOL52a/license/license.dat:
FlexNet Licensing error:-96,7
* Reason: The license servers cannot be reached at all. The <code>ping</code> or <code>host</code> tests mentioned above fail.
* Solution: Review the client license file and DNS lookup domains.

=== License file does not support this version ===
* Error details:
Feature: SERIAL
'''Application version > License version: 5.21 > 3.5'''
License path: /Users/stern/Applications/COMSOL52a/license/license.dat:
FlexNet Licensing error:-21,126
* Reason: Only the first of two steps for connecting to the license servers succeeds, which may happen when the redundant license service runs on the alternate server instead of the one you're connected to.
* Solution: When using SSH, try <code>clicense2</code> instead of <code>clicense1</code>, or vice versa. Revisit your <code>~/.ssh/config</code> file or use the alternate command line as shown in the SSH section above.

== Multiple CPU cores ==
Ways to start multi-core (from http://www.comsol.com/support/knowledgebase/1096/)
comsol -np 8

export COMSOL_NUM_THREADS=8 # automatically provided on ''Carbon''
comsol
* COMSOL does not benefit from hyperthreading (oversubscribing the physical core count).

== Batch jobs ==

* http://www.comsol.com/support/knowledgebase/1001/ .
* [[Media:HPC 2012-05-16 Comsol parallel usage.pdf]] (Modified from http://www.comsol.com/support/knowledgebase/1001/files/4027/cluster_install_linux_42a.pptx – Appendix slide, PBS)
* '''Note:''' MPD is not supported on Carbon (I patched the main comsol script), and no longer necessary at all in Comsol-4.3.

=== Sample Job file ===
See:
$COMSOL_HOME/sample.job

== Important COMSOL Support Articles ==
* Toplevel suport page: https://www.comsol.com/support

=== KB Articles ===
* [https://www.comsol.com/support/knowledgebase/1001 Running COMSOL® in parallel on clusters]
* [https://www.comsol.com/support/knowledgebase/1250 Running parametric sweeps, batch sweeps, and cluster sweeps from the command line]
* [https://www.comsol.com/support/knowledgebase/1255 Reducing the amount of solution data stored in a model]
* [https://www.comsol.com/support/knowledgebase/830 Handling large models]

=== Blog-style articles ===
* [https://www.comsol.com/blogs/the-power-of-the-batch-sweep/ The Power of the Batch Sweep]
* [https://www.comsol.com/blogs/how-to-run-simulations-in-batch-mode-from-the-command-line/ How to Run Simulations in Batch Mode from the Command Line]

HPC/Applications/comsol

2026-03-04T18:24:27Z

Stern: /* Running the desktop application */

== Introduction ==
The [http://www.comsol.com/products/multiphysics/ '''COMSOL Multiphysics'''] application suite
is available for Mac, Windows, and Linux.

== Access ==
[[Image:HPC Comsol licensed modules.png|right|450px|thumb|COMSOL features licensed at CNM as of 2024-10.]]
=== COMSOL Multiphysics ===
Access to '''COMSOL Multiphysics''', the main application with both a graphical user interface and batch computing capabilities, will be granted '''''upon request''''' .

You must be:
* An '''''Argonne employee'''''  in one of the following roles:
** '''''NST staff member'''''  (including PostDocs, joint appointments, and students at NST).
** '''''CNM Facility User'''''  (including ''non-NST Argonne employees''), with an active proposal that has COMSOL in scope. Please see [[HPC/FAQ#Access_and_proposals]].

The set of available COMSOL modules that we have licensed is shown at right.

* The native desktop version can be installed only on ''Argonne-owned'' computers (laptop or desktop). Please contact [mailto:help@anl.gov?Subject=COMSOL%20license%20access%20request CNM support]. Make sure to review [http://www.comsol.com/support/updates/ Product Updates].
* To run COMSOL on our HPC system ''Carbon'', as GUI or as batch computing jobs, your proposal must have ''Carbon'' access in scope.

=== COMSOL Server ===
''CNM Facility Users'' who are not Argonne employees are not eligible to access
the full '''COMSOL Multiphysics''' product, in compliance with its licensing terms and conditions.
However, the [https://www.comsol.com/comsol-server '''COMSOL Server'''] product offers an alternative
albeit more narrow means of access.

All users, including '''''non-Argonne users''''' may be able to run, over a web interface in their browser,
pre-parameterized "fixed-model" simulations that were made with the '''COMSOL Application Builder''' (requires Windows).

To inquire about using COMSOL server applications,
please contact us.

We will review possible purchase paths for COMSOL Server, and the ''considerable effort'' likely required by users and the CNM to develop and deploy a COMSOL application.

== Support ==
* For questions on how to run or start COMSOL:
*: Submit an [[HPC/Support | '''IT support request''']]
* For questions related to most COMSOL data or scripts, contact COMSOL support at:
*: [mailto:support@comsol.com '''support'''@comsol.com]
* '''Exception:''' If your support question might contain data that is subject to '''US Export Control''' regulations, contact instead:
*: [mailto:us-export-control@us.comsol.com '''us-export-control'''@'''us'''.comsol.com]

== The modes to run COMSOL ==
There are several ways of running COMSOL.

=== Running the desktop application ===
To edit models and analyze results, use the COMSOL graphical user interface (GUI). You can run it in several ways, each explained in more detail in the following sections.

# Run on your Argonne-provided computer, checking out Carbon's license remotely.
#* Requires a full installation of COMSOL on your desktoo or laptop computer.
#* You will require access to the COMSOL license server running on Carbon, either through VPN (fairly easy) or SSH tunneling (can be intimidating for first-time users). See below for details.
# Run on Carbon, display on your desktop using the [http://en.wikipedia.org/wiki/X_Window_System X Window system (X11)].
#* Requires an ''X11 Server'' application running on your client machine, and SSH X11 forwarding.
#* The X11 protocol can be slow over LAN.
# Run on Carbon in a [[HPC/Network Access/Virtual Desktop | virtual desktop environment (VNC)]], viewed in a VNC client on your machine.
#* Requires a VNC client on your machine.
#* It can be confusing where VNC and the app runs.
#* The virtual desktop presents a (deliberatly) sparse windowing environment – Copy&Paste can be difficult to use.

=== Running a batch calculation ===
When the computational stage of models takes longer than a few minutes to complete, it will be beneficial to submit save the model file and run it in parallel on Carbon.
See the [[#Batch jobs|section below]].

== Use with X11 ==
For casual use, and with [[HPC/Network Access/SSH Tunnel Setup on Linux and MacOS | X11 tunneling to Carbon]] set up, simply add to your <code>~/.bashrc</code> file on Carbon:
module load comsol

Then, from the command line on a Carbon login node, start the COMSOL GUI:
comsol &

License configuration is automatic on Carbon.

Comsol will use a number of cores to run calculations multithreaded by default. (On Carbon, COMSOL_NUM_THREADS is set.)

'''Quit the application''' as soon as you are done. Only one user can use Comsol at a time.

== Use with VNC ==
See general [[HPC/Network Access/Virtual Desktop | instructions on using VNC]], and the COMSOL example therein.

== Native desktop version ==

To get the fastest response from the COMSOL GUI, run a native COMSOL version on your desktop machine or laptop, called ''client machine'' below.

Enter license details either in the COMSOL Setup configuration dialog.

== Native desktop version — License configuration ==
A client's COMSOL license needs to be configured depending on how the client reaches CNM network resources. Follow the appropriate section:

* [[#License configuration when inside CNM or on CNM VPN|License configuration when '''inside CNM''' or on '''CNM VPN''']]
: Requires that the user's machine is logically within the network of CNM/NST, either by a wired/auth WiFi connection in the building or by VPN under a CNM/NST profile.
* [[#License configuration when outside CNM – requires SSH Tunneling|License configuration when '''outside CNM''' – requires SSH Tunneling]]
: ''Non-NST staff'' usually have a VPN profile that places the machine logically [[#License configuration when outside CNM – requires SSH Tunneling|outside of NST's network.]]

== License configuration when inside CNM or on CNM VPN ==
=== Ensure correct DNS search domains ===
* Inspect the client machine's network configuration.
* Ensure that '''two''' entries are listed as DNS Search Domains for the current network profile:
nst.anl.gov
cnm.anl.gov
: (The order of entries does not matter for the purposes considered here.)
* If the client machine is ''wired to CNM networks'', the domains are normally provided correctly by DHCP.
* If the client machine is ''connected over VPN'', you may need edit the respective network profile.
:* On a Mac, find the VPN configuration details under System Preferences, Network, Advanced… . You may need to unlock the Network panel first before you can make changes under "Advanced…". It appears that those changes are being saved and will be activated automatically the next time that this VPN session gets started.
: [[Image:VNC_add_cnm_to_DNS_1.png|400px]] [[Image:VNC_add_cnm_to_DNS_2.png|400px]]
:* On Windows or Linux machines, take ''analogous'' steps, using the platform's own configuration tools.

* Exclude license manager components from client desktop installation. (Those are all handled server-side.)

=== License format ===
* Floating Network License (FNL), "Three-server redundancy"

=== License servers ===
* On client machines, while running the COMSOL installer or license configuration, specify license servers and ports as follows:
1718 clicense1
1718 clicense2
1718 clicense3
: '''Note:''' Use ''short host names only''. (Technical Reason: The license must be usable both on Carbon's internal network as well as on divisional networks).
1718 clicense1<strike>.cnm.anl.gov</strike>
1718 clicense2<strike>.cnm.anl.gov</strike>
1718 clicense3<strike>.cnm.anl.gov</strike>

'''Advanced:''' A client machine's COMSOL license file <code><COMSOLDIR>/Multiphysics/license/license.dat</code> should look as follows:
<source lang="make">
SERVER clicense1 ANY 1718
SERVER clicense2 ANY 1718
SERVER clicense3 ANY 1718
USE_SERVER
</source>

<code><COMSOLDIR></code> is located by default at:
* '''Windows''': C:\Program files\COMSOL\COMSOL''nn''
* '''Linux''': /usr/local/comsol''nn''
* '''Mac''': /Applications/COMSOL''nn''
("''nn''" is the application's abbreviated version number.)

=== Network Troubleshooting ===
When using the license access model descibed here (as opposed to tunneled over SSH), ensure the following:
* Verify that short host names can be resolved to IP addresses. Either one of the following commands should produce sensible output:
host clicense1
: or:
ping -c 3 clicense1
* Verify that ''both license ports on the same server'' can be reached from your client machine:
telnet '''clicense1''' 1718
telnet '''clicense1''' 1719
: or:
telnet '''clicense2''' 1718
telnet '''clicense2''' 1719
: A ''successful'' connection looks like:
Trying 146.139.x.y...
Connected to clicense1.cnm.anl.gov.
Escape character is '^]'.
:: Do not enter any data. Instead, close each connection by hitting <code>Ctrl</code><code>]</code> (closing square bracket), followed by <code>Ctrl</code><code>D</code>.
: A ''failed'' connection gives one of these errors:
telnet: connect to address 146.139.x.y: Operation timed out
: or:
telnet: connect to address 146.139.x.y: Connection refused
:: In these cases, ask that your IP firewall configuration be reviewed and changed.

== License configuration when outside CNM – requires SSH Tunneling ==
This section documents how to run COMSOL from a network location outside CNM.

As a one-time preparation step, two items must be configured:
* The license host that COMSOL is to contact, and
* An SSH connection that will allow your machine to access the COMSOL license server inside CNM's network.

To then ''run'' the COMSOL application, the configured SSH connection must be up and kept running for the entire time that you run the COMSOL. When you exit COMSOL and wish to start it again, you can keep the SSH connection running, it will be re-used for the next COMSOL run.

=== Configure COMSOL ===
The following needs to be done just once, typically during COMSOL installation:

In the COMSOL Installer, or in COMSOL Preferences, enter:
* '''License format:''' Floating Network License (FNL), "Port number and Hostname"
:: Do ''not use'' "Three-server redundancy", since it cannot (easily) be accessed over SSH. Configure with just one server, and in the rare situation of a server failover, temporarily tweak your SSH configuration to contact the alternate server (see below).
* '''Server:''' <code>127.0.0.1</code>
:: Do not use "localhost", as COMSOL interprets this incorrectly to mean that your machine ''is'' a license server.

* '''Port:''' <code>1718</code>


=== Establish connection to CNM license server ===
This step needs to be done prior to running COMSOL in your desktop session:
* Connect to <code>mega</code> with the following command in a terminal window:
ssh -L 1718:'''clicense1''':1718 -L 1719:'''clicense1''':1719 mega.cnm.anl.gov
* You can leave this connection running in your terminal window across several invocations of COMSOL.

=== Run COMSOL ===
# Ensure your connection to "mega" is still running. If not, restart it.
# Start the COMSOL application.

=== Troubleshooting ===
* If you get an error that the number of licenses is "exhausted", that is confirmation that you correctly reached the license server, though other users are using licenses.
* If the license access attempt takes a long time or hangs, the service might temporarily run on a secondary server. In this case, stop the mega session and restart it, connecting to the alternate license server:
ssh -L 1718:clicense'''2''':1718 -L 1719:clicense'''2''':1719 mega.cnm.anl.gov

== General License Troubleshooting ==
The following are typical error messages.
=== Could not obtain license for COMSOL Multiphysics GUI ===
* Error details:
License error: -4.
'''Licensed number of users already reached.'''
Feature: COMSOLGUI
License path: /Users/stern/Applications/COMSOL52a/license/license.dat:
FlexNet Licensing error:-4,132
* Reason: The connection to the license server succeeded, but the license token(s) you requested are in use.
* Solution: Wait a bit, or [http://carbonmgmt/ganglia/lic.php identify who's using COMSOL] and politely ask the user when they might be finished with their work.

=== License server machine is down or not responding ===
* Error details:
License error: -96.
See the system administrator about starting the license server system, or
make sure you're referring to the right host (see LM_LICENSE_FILE).
Feature: SERIAL
Hostname: clicense3
License path: /Users/stern/Applications/COMSOL52a/license/license.dat:
FlexNet Licensing error:-96,7
* Reason: The license servers cannot be reached at all. The <code>ping</code> or <code>host</code> tests mentioned above fail.
* Solution: Review the client license file and DNS lookup domains.

=== License file does not support this version ===
* Error details:
Feature: SERIAL
'''Application version > License version: 5.21 > 3.5'''
License path: /Users/stern/Applications/COMSOL52a/license/license.dat:
FlexNet Licensing error:-21,126
* Reason: Only the first of two steps for connecting to the license servers succeeds, which may happen when the redundant license service runs on the alternate server instead of the one you're connected to.
* Solution: When using SSH, try <code>clicense2</code> instead of <code>clicense1</code>, or vice versa. Revisit your <code>~/.ssh/config</code> file or use the alternate command line as shown in the SSH section above.

== Multiple CPU cores ==
Ways to start multi-core (from http://www.comsol.com/support/knowledgebase/1096/)
comsol -np 8

export COMSOL_NUM_THREADS=8 # automatically provided on ''Carbon''
comsol
* COMSOL does not benefit from hyperthreading (oversubscribing the physical core count).

== Batch jobs ==

* http://www.comsol.com/support/knowledgebase/1001/ .
* [[Media:HPC 2012-05-16 Comsol parallel usage.pdf]] (Modified from http://www.comsol.com/support/knowledgebase/1001/files/4027/cluster_install_linux_42a.pptx – Appendix slide, PBS)
* '''Note:''' MPD is not supported on Carbon (I patched the main comsol script), and no longer necessary at all in Comsol-4.3.

=== Sample Job file ===
See:
$COMSOL_HOME/sample.job

== Important COMSOL Support Articles ==
* Toplevel suport page: https://www.comsol.com/support

=== KB Articles ===
* [https://www.comsol.com/support/knowledgebase/1001 Running COMSOL® in parallel on clusters]
* [https://www.comsol.com/support/knowledgebase/1250 Running parametric sweeps, batch sweeps, and cluster sweeps from the command line]
* [https://www.comsol.com/support/knowledgebase/1255 Reducing the amount of solution data stored in a model]
* [https://www.comsol.com/support/knowledgebase/830 Handling large models]

=== Blog-style articles ===
* [https://www.comsol.com/blogs/the-power-of-the-batch-sweep/ The Power of the Batch Sweep]
* [https://www.comsol.com/blogs/how-to-run-simulations-in-batch-mode-from-the-command-line/ How to Run Simulations in Batch Mode from the Command Line]

HPC/Applications/comsol

2026-02-26T17:17:36Z

Stern: /* License Troubleshooting */

== Introduction ==
The [http://www.comsol.com/products/multiphysics/ '''COMSOL Multiphysics'''] application suite
is available for Mac, Windows, and Linux.

== Access ==
[[Image:HPC Comsol licensed modules.png|right|450px|thumb|COMSOL features licensed at CNM as of 2024-10.]]
=== COMSOL Multiphysics ===
Access to '''COMSOL Multiphysics''', the main application with both a graphical user interface and batch computing capabilities, will be granted '''''upon request''''' .

You must be:
* An '''''Argonne employee'''''  in one of the following roles:
** '''''NST staff member'''''  (including PostDocs, joint appointments, and students at NST).
** '''''CNM Facility User'''''  (including ''non-NST Argonne employees''), with an active proposal that has COMSOL in scope. Please see [[HPC/FAQ#Access_and_proposals]].

The set of available COMSOL modules that we have licensed is shown at right.

* The native desktop version can be installed only on ''Argonne-owned'' computers (laptop or desktop). Please contact [mailto:help@anl.gov?Subject=COMSOL%20license%20access%20request CNM support]. Make sure to review [http://www.comsol.com/support/updates/ Product Updates].
* To run COMSOL on our HPC system ''Carbon'', as GUI or as batch computing jobs, your proposal must have ''Carbon'' access in scope.

=== COMSOL Server ===
''CNM Facility Users'' who are not Argonne employees are not eligible to access
the full '''COMSOL Multiphysics''' product, in compliance with its licensing terms and conditions.
However, the [https://www.comsol.com/comsol-server '''COMSOL Server'''] product offers an alternative
albeit more narrow means of access.

All users, including '''''non-Argonne users''''' may be able to run, over a web interface in their browser,
pre-parameterized "fixed-model" simulations that were made with the '''COMSOL Application Builder''' (requires Windows).

To inquire about using COMSOL server applications,
please contact us.

We will review possible purchase paths for COMSOL Server, and the ''considerable effort'' likely required by users and the CNM to develop and deploy a COMSOL application.

== Support ==
* For questions on how to run or start COMSOL:
*: Submit an [[HPC/Support | '''IT support request''']]
* For questions related to most COMSOL data or scripts, contact COMSOL support at:
*: [mailto:support@comsol.com '''support'''@comsol.com]
* '''Exception:''' If your support question might contain data that is subject to '''US Export Control''' regulations, contact instead:
*: [mailto:us-export-control@us.comsol.com '''us-export-control'''@'''us'''.comsol.com]

== The modes to run COMSOL ==
There are several ways of running COMSOL.

=== Running the desktop application ===
To edit models and analyze results, use the COMSOL graphical user interface (GUI). You can run it in several ways, each explained in more detail in the following sections.

# Run on Carbon, display on your desktop using the [http://en.wikipedia.org/wiki/X_Window_System X Window system (X11)].
#* Requires an ''X11 Server'' application running on your client machine, and SSH X11 forwarding.
#* The X11 protocol can be slow over LAN.
# Run on Carbon within a [[HPC/Network Access/Virtual Desktop | virtual desktop environment (VNC)]].
#* Requires a VNC client on your machine.
#* It can be confusing where VNC and the app runs.
#* The virtual desktop presents a (deliberatly) sparse windowing environment – Copy&Paste can be difficult to use.
# Run on your Argonne-provided computer, checking out Carbon's license remotely.
#* Requires a full installation of COMSOL on your desktoo or laptop computer.
#* You will require access to the COMSOL license server running on Carbon, either through VPN (fairly easy) or SSH tunneling (can be intimidating for first-time users). See below for details.

=== Running a batch calculation ===
When the computational stage of models takes longer than a few minutes to complete, it will be beneficial to submit save the model file and run it in parallel on Carbon.
See the [[#Batch jobs|section below]].

== Use with X11 ==
For casual use, and with [[HPC/Network Access/SSH Tunnel Setup on Linux and MacOS | X11 tunneling to Carbon]] set up, simply add to your <code>~/.bashrc</code> file on Carbon:
module load comsol

Then, from the command line on a Carbon login node, start the COMSOL GUI:
comsol &

License configuration is automatic on Carbon.

Comsol will use a number of cores to run calculations multithreaded by default. (On Carbon, COMSOL_NUM_THREADS is set.)

'''Quit the application''' as soon as you are done. Only one user can use Comsol at a time.

== Use with VNC ==
See general [[HPC/Network Access/Virtual Desktop | instructions on using VNC]], and the COMSOL example therein.

== Native desktop version ==

To get the fastest response from the COMSOL GUI, run a native COMSOL version on your desktop machine or laptop, called ''client machine'' below.

Enter license details either in the COMSOL Setup configuration dialog.

== Native desktop version — License configuration ==
A client's COMSOL license needs to be configured depending on how the client reaches CNM network resources. Follow the appropriate section:

* [[#License configuration when inside CNM or on CNM VPN|License configuration when '''inside CNM''' or on '''CNM VPN''']]
: Requires that the user's machine is logically within the network of CNM/NST, either by a wired/auth WiFi connection in the building or by VPN under a CNM/NST profile.
* [[#License configuration when outside CNM – requires SSH Tunneling|License configuration when '''outside CNM''' – requires SSH Tunneling]]
: ''Non-NST staff'' usually have a VPN profile that places the machine logically [[#License configuration when outside CNM – requires SSH Tunneling|outside of NST's network.]]

== License configuration when inside CNM or on CNM VPN ==
=== Ensure correct DNS search domains ===
* Inspect the client machine's network configuration.
* Ensure that '''two''' entries are listed as DNS Search Domains for the current network profile:
nst.anl.gov
cnm.anl.gov
: (The order of entries does not matter for the purposes considered here.)
* If the client machine is ''wired to CNM networks'', the domains are normally provided correctly by DHCP.
* If the client machine is ''connected over VPN'', you may need edit the respective network profile.
:* On a Mac, find the VPN configuration details under System Preferences, Network, Advanced… . You may need to unlock the Network panel first before you can make changes under "Advanced…". It appears that those changes are being saved and will be activated automatically the next time that this VPN session gets started.
: [[Image:VNC_add_cnm_to_DNS_1.png|400px]] [[Image:VNC_add_cnm_to_DNS_2.png|400px]]
:* On Windows or Linux machines, take ''analogous'' steps, using the platform's own configuration tools.

* Exclude license manager components from client desktop installation. (Those are all handled server-side.)

=== License format ===
* Floating Network License (FNL), "Three-server redundancy"

=== License servers ===
* On client machines, while running the COMSOL installer or license configuration, specify license servers and ports as follows:
1718 clicense1
1718 clicense2
1718 clicense3
: '''Note:''' Use ''short host names only''. (Technical Reason: The license must be usable both on Carbon's internal network as well as on divisional networks).
1718 clicense1<strike>.cnm.anl.gov</strike>
1718 clicense2<strike>.cnm.anl.gov</strike>
1718 clicense3<strike>.cnm.anl.gov</strike>

'''Advanced:''' A client machine's COMSOL license file <code><COMSOLDIR>/Multiphysics/license/license.dat</code> should look as follows:
<source lang="make">
SERVER clicense1 ANY 1718
SERVER clicense2 ANY 1718
SERVER clicense3 ANY 1718
USE_SERVER
</source>

<code><COMSOLDIR></code> is located by default at:
* '''Windows''': C:\Program files\COMSOL\COMSOL''nn''
* '''Linux''': /usr/local/comsol''nn''
* '''Mac''': /Applications/COMSOL''nn''
("''nn''" is the application's abbreviated version number.)

=== Network Troubleshooting ===
When using the license access model descibed here (as opposed to tunneled over SSH), ensure the following:
* Verify that short host names can be resolved to IP addresses. Either one of the following commands should produce sensible output:
host clicense1
: or:
ping -c 3 clicense1
* Verify that ''both license ports on the same server'' can be reached from your client machine:
telnet '''clicense1''' 1718
telnet '''clicense1''' 1719
: or:
telnet '''clicense2''' 1718
telnet '''clicense2''' 1719
: A ''successful'' connection looks like:
Trying 146.139.x.y...
Connected to clicense1.cnm.anl.gov.
Escape character is '^]'.
:: Do not enter any data. Instead, close each connection by hitting <code>Ctrl</code><code>]</code> (closing square bracket), followed by <code>Ctrl</code><code>D</code>.
: A ''failed'' connection gives one of these errors:
telnet: connect to address 146.139.x.y: Operation timed out
: or:
telnet: connect to address 146.139.x.y: Connection refused
:: In these cases, ask that your IP firewall configuration be reviewed and changed.

== License configuration when outside CNM – requires SSH Tunneling ==
This section documents how to run COMSOL from a network location outside CNM.

As a one-time preparation step, two items must be configured:
* The license host that COMSOL is to contact, and
* An SSH connection that will allow your machine to access the COMSOL license server inside CNM's network.

To then ''run'' the COMSOL application, the configured SSH connection must be up and kept running for the entire time that you run the COMSOL. When you exit COMSOL and wish to start it again, you can keep the SSH connection running, it will be re-used for the next COMSOL run.

=== Configure COMSOL ===
The following needs to be done just once, typically during COMSOL installation:

In the COMSOL Installer, or in COMSOL Preferences, enter:
* '''License format:''' Floating Network License (FNL), "Port number and Hostname"
:: Do ''not use'' "Three-server redundancy", since it cannot (easily) be accessed over SSH. Configure with just one server, and in the rare situation of a server failover, temporarily tweak your SSH configuration to contact the alternate server (see below).
* '''Server:''' <code>127.0.0.1</code>
:: Do not use "localhost", as COMSOL interprets this incorrectly to mean that your machine ''is'' a license server.

* '''Port:''' <code>1718</code>


=== Establish connection to CNM license server ===
This step needs to be done prior to running COMSOL in your desktop session:
* Connect to <code>mega</code> with the following command in a terminal window:
ssh -L 1718:'''clicense1''':1718 -L 1719:'''clicense1''':1719 mega.cnm.anl.gov
* You can leave this connection running in your terminal window across several invocations of COMSOL.

=== Run COMSOL ===
# Ensure your connection to "mega" is still running. If not, restart it.
# Start the COMSOL application.

=== Troubleshooting ===
* If you get an error that the number of licenses is "exhausted", that is confirmation that you correctly reached the license server, though other users are using licenses.
* If the license access attempt takes a long time or hangs, the service might temporarily run on a secondary server. In this case, stop the mega session and restart it, connecting to the alternate license server:
ssh -L 1718:clicense'''2''':1718 -L 1719:clicense'''2''':1719 mega.cnm.anl.gov

== General License Troubleshooting ==
The following are typical error messages.
=== Could not obtain license for COMSOL Multiphysics GUI ===
* Error details:
License error: -4.
'''Licensed number of users already reached.'''
Feature: COMSOLGUI
License path: /Users/stern/Applications/COMSOL52a/license/license.dat:
FlexNet Licensing error:-4,132
* Reason: The connection to the license server succeeded, but the license token(s) you requested are in use.
* Solution: Wait a bit, or [http://carbonmgmt/ganglia/lic.php identify who's using COMSOL] and politely ask the user when they might be finished with their work.

=== License server machine is down or not responding ===
* Error details:
License error: -96.
See the system administrator about starting the license server system, or
make sure you're referring to the right host (see LM_LICENSE_FILE).
Feature: SERIAL
Hostname: clicense3
License path: /Users/stern/Applications/COMSOL52a/license/license.dat:
FlexNet Licensing error:-96,7
* Reason: The license servers cannot be reached at all. The <code>ping</code> or <code>host</code> tests mentioned above fail.
* Solution: Review the client license file and DNS lookup domains.

=== License file does not support this version ===
* Error details:
Feature: SERIAL
'''Application version > License version: 5.21 > 3.5'''
License path: /Users/stern/Applications/COMSOL52a/license/license.dat:
FlexNet Licensing error:-21,126
* Reason: Only the first of two steps for connecting to the license servers succeeds, which may happen when the redundant license service runs on the alternate server instead of the one you're connected to.
* Solution: When using SSH, try <code>clicense2</code> instead of <code>clicense1</code>, or vice versa. Revisit your <code>~/.ssh/config</code> file or use the alternate command line as shown in the SSH section above.

== Multiple CPU cores ==
Ways to start multi-core (from http://www.comsol.com/support/knowledgebase/1096/)
comsol -np 8

export COMSOL_NUM_THREADS=8 # automatically provided on ''Carbon''
comsol
* COMSOL does not benefit from hyperthreading (oversubscribing the physical core count).

== Batch jobs ==

* http://www.comsol.com/support/knowledgebase/1001/ .
* [[Media:HPC 2012-05-16 Comsol parallel usage.pdf]] (Modified from http://www.comsol.com/support/knowledgebase/1001/files/4027/cluster_install_linux_42a.pptx – Appendix slide, PBS)
* '''Note:''' MPD is not supported on Carbon (I patched the main comsol script), and no longer necessary at all in Comsol-4.3.

=== Sample Job file ===
See:
$COMSOL_HOME/sample.job

== Important COMSOL Support Articles ==
* Toplevel suport page: https://www.comsol.com/support

=== KB Articles ===
* [https://www.comsol.com/support/knowledgebase/1001 Running COMSOL® in parallel on clusters]
* [https://www.comsol.com/support/knowledgebase/1250 Running parametric sweeps, batch sweeps, and cluster sweeps from the command line]
* [https://www.comsol.com/support/knowledgebase/1255 Reducing the amount of solution data stored in a model]
* [https://www.comsol.com/support/knowledgebase/830 Handling large models]

=== Blog-style articles ===
* [https://www.comsol.com/blogs/the-power-of-the-batch-sweep/ The Power of the Batch Sweep]
* [https://www.comsol.com/blogs/how-to-run-simulations-in-batch-mode-from-the-command-line/ How to Run Simulations in Batch Mode from the Command Line]

HPC/Applications/comsol

2026-02-26T17:17:10Z

Stern: /* Establish connection to CNM license server */

== Introduction ==
The [http://www.comsol.com/products/multiphysics/ '''COMSOL Multiphysics'''] application suite
is available for Mac, Windows, and Linux.

== Access ==
[[Image:HPC Comsol licensed modules.png|right|450px|thumb|COMSOL features licensed at CNM as of 2024-10.]]
=== COMSOL Multiphysics ===
Access to '''COMSOL Multiphysics''', the main application with both a graphical user interface and batch computing capabilities, will be granted '''''upon request''''' .

You must be:
* An '''''Argonne employee'''''  in one of the following roles:
** '''''NST staff member'''''  (including PostDocs, joint appointments, and students at NST).
** '''''CNM Facility User'''''  (including ''non-NST Argonne employees''), with an active proposal that has COMSOL in scope. Please see [[HPC/FAQ#Access_and_proposals]].

The set of available COMSOL modules that we have licensed is shown at right.

* The native desktop version can be installed only on ''Argonne-owned'' computers (laptop or desktop). Please contact [mailto:help@anl.gov?Subject=COMSOL%20license%20access%20request CNM support]. Make sure to review [http://www.comsol.com/support/updates/ Product Updates].
* To run COMSOL on our HPC system ''Carbon'', as GUI or as batch computing jobs, your proposal must have ''Carbon'' access in scope.

=== COMSOL Server ===
''CNM Facility Users'' who are not Argonne employees are not eligible to access
the full '''COMSOL Multiphysics''' product, in compliance with its licensing terms and conditions.
However, the [https://www.comsol.com/comsol-server '''COMSOL Server'''] product offers an alternative
albeit more narrow means of access.

All users, including '''''non-Argonne users''''' may be able to run, over a web interface in their browser,
pre-parameterized "fixed-model" simulations that were made with the '''COMSOL Application Builder''' (requires Windows).

To inquire about using COMSOL server applications,
please contact us.

We will review possible purchase paths for COMSOL Server, and the ''considerable effort'' likely required by users and the CNM to develop and deploy a COMSOL application.

== Support ==
* For questions on how to run or start COMSOL:
*: Submit an [[HPC/Support | '''IT support request''']]
* For questions related to most COMSOL data or scripts, contact COMSOL support at:
*: [mailto:support@comsol.com '''support'''@comsol.com]
* '''Exception:''' If your support question might contain data that is subject to '''US Export Control''' regulations, contact instead:
*: [mailto:us-export-control@us.comsol.com '''us-export-control'''@'''us'''.comsol.com]

== The modes to run COMSOL ==
There are several ways of running COMSOL.

=== Running the desktop application ===
To edit models and analyze results, use the COMSOL graphical user interface (GUI). You can run it in several ways, each explained in more detail in the following sections.

# Run on Carbon, display on your desktop using the [http://en.wikipedia.org/wiki/X_Window_System X Window system (X11)].
#* Requires an ''X11 Server'' application running on your client machine, and SSH X11 forwarding.
#* The X11 protocol can be slow over LAN.
# Run on Carbon within a [[HPC/Network Access/Virtual Desktop | virtual desktop environment (VNC)]].
#* Requires a VNC client on your machine.
#* It can be confusing where VNC and the app runs.
#* The virtual desktop presents a (deliberatly) sparse windowing environment – Copy&Paste can be difficult to use.
# Run on your Argonne-provided computer, checking out Carbon's license remotely.
#* Requires a full installation of COMSOL on your desktoo or laptop computer.
#* You will require access to the COMSOL license server running on Carbon, either through VPN (fairly easy) or SSH tunneling (can be intimidating for first-time users). See below for details.

=== Running a batch calculation ===
When the computational stage of models takes longer than a few minutes to complete, it will be beneficial to submit save the model file and run it in parallel on Carbon.
See the [[#Batch jobs|section below]].

== Use with X11 ==
For casual use, and with [[HPC/Network Access/SSH Tunnel Setup on Linux and MacOS | X11 tunneling to Carbon]] set up, simply add to your <code>~/.bashrc</code> file on Carbon:
module load comsol

Then, from the command line on a Carbon login node, start the COMSOL GUI:
comsol &

License configuration is automatic on Carbon.

Comsol will use a number of cores to run calculations multithreaded by default. (On Carbon, COMSOL_NUM_THREADS is set.)

'''Quit the application''' as soon as you are done. Only one user can use Comsol at a time.

== Use with VNC ==
See general [[HPC/Network Access/Virtual Desktop | instructions on using VNC]], and the COMSOL example therein.

== Native desktop version ==

To get the fastest response from the COMSOL GUI, run a native COMSOL version on your desktop machine or laptop, called ''client machine'' below.

Enter license details either in the COMSOL Setup configuration dialog.

== Native desktop version — License configuration ==
A client's COMSOL license needs to be configured depending on how the client reaches CNM network resources. Follow the appropriate section:

* [[#License configuration when inside CNM or on CNM VPN|License configuration when '''inside CNM''' or on '''CNM VPN''']]
: Requires that the user's machine is logically within the network of CNM/NST, either by a wired/auth WiFi connection in the building or by VPN under a CNM/NST profile.
* [[#License configuration when outside CNM – requires SSH Tunneling|License configuration when '''outside CNM''' – requires SSH Tunneling]]
: ''Non-NST staff'' usually have a VPN profile that places the machine logically [[#License configuration when outside CNM – requires SSH Tunneling|outside of NST's network.]]

== License configuration when inside CNM or on CNM VPN ==
=== Ensure correct DNS search domains ===
* Inspect the client machine's network configuration.
* Ensure that '''two''' entries are listed as DNS Search Domains for the current network profile:
nst.anl.gov
cnm.anl.gov
: (The order of entries does not matter for the purposes considered here.)
* If the client machine is ''wired to CNM networks'', the domains are normally provided correctly by DHCP.
* If the client machine is ''connected over VPN'', you may need edit the respective network profile.
:* On a Mac, find the VPN configuration details under System Preferences, Network, Advanced… . You may need to unlock the Network panel first before you can make changes under "Advanced…". It appears that those changes are being saved and will be activated automatically the next time that this VPN session gets started.
: [[Image:VNC_add_cnm_to_DNS_1.png|400px]] [[Image:VNC_add_cnm_to_DNS_2.png|400px]]
:* On Windows or Linux machines, take ''analogous'' steps, using the platform's own configuration tools.

* Exclude license manager components from client desktop installation. (Those are all handled server-side.)

=== License format ===
* Floating Network License (FNL), "Three-server redundancy"

=== License servers ===
* On client machines, while running the COMSOL installer or license configuration, specify license servers and ports as follows:
1718 clicense1
1718 clicense2
1718 clicense3
: '''Note:''' Use ''short host names only''. (Technical Reason: The license must be usable both on Carbon's internal network as well as on divisional networks).
1718 clicense1<strike>.cnm.anl.gov</strike>
1718 clicense2<strike>.cnm.anl.gov</strike>
1718 clicense3<strike>.cnm.anl.gov</strike>

'''Advanced:''' A client machine's COMSOL license file <code><COMSOLDIR>/Multiphysics/license/license.dat</code> should look as follows:
<source lang="make">
SERVER clicense1 ANY 1718
SERVER clicense2 ANY 1718
SERVER clicense3 ANY 1718
USE_SERVER
</source>

<code><COMSOLDIR></code> is located by default at:
* '''Windows''': C:\Program files\COMSOL\COMSOL''nn''
* '''Linux''': /usr/local/comsol''nn''
* '''Mac''': /Applications/COMSOL''nn''
("''nn''" is the application's abbreviated version number.)

=== Network Troubleshooting ===
When using the license access model descibed here (as opposed to tunneled over SSH), ensure the following:
* Verify that short host names can be resolved to IP addresses. Either one of the following commands should produce sensible output:
host clicense1
: or:
ping -c 3 clicense1
* Verify that ''both license ports on the same server'' can be reached from your client machine:
telnet '''clicense1''' 1718
telnet '''clicense1''' 1719
: or:
telnet '''clicense2''' 1718
telnet '''clicense2''' 1719
: A ''successful'' connection looks like:
Trying 146.139.x.y...
Connected to clicense1.cnm.anl.gov.
Escape character is '^]'.
:: Do not enter any data. Instead, close each connection by hitting <code>Ctrl</code><code>]</code> (closing square bracket), followed by <code>Ctrl</code><code>D</code>.
: A ''failed'' connection gives one of these errors:
telnet: connect to address 146.139.x.y: Operation timed out
: or:
telnet: connect to address 146.139.x.y: Connection refused
:: In these cases, ask that your IP firewall configuration be reviewed and changed.

== License configuration when outside CNM – requires SSH Tunneling ==
This section documents how to run COMSOL from a network location outside CNM.

As a one-time preparation step, two items must be configured:
* The license host that COMSOL is to contact, and
* An SSH connection that will allow your machine to access the COMSOL license server inside CNM's network.

To then ''run'' the COMSOL application, the configured SSH connection must be up and kept running for the entire time that you run the COMSOL. When you exit COMSOL and wish to start it again, you can keep the SSH connection running, it will be re-used for the next COMSOL run.

=== Configure COMSOL ===
The following needs to be done just once, typically during COMSOL installation:

In the COMSOL Installer, or in COMSOL Preferences, enter:
* '''License format:''' Floating Network License (FNL), "Port number and Hostname"
:: Do ''not use'' "Three-server redundancy", since it cannot (easily) be accessed over SSH. Configure with just one server, and in the rare situation of a server failover, temporarily tweak your SSH configuration to contact the alternate server (see below).
* '''Server:''' <code>127.0.0.1</code>
:: Do not use "localhost", as COMSOL interprets this incorrectly to mean that your machine ''is'' a license server.

* '''Port:''' <code>1718</code>


=== Establish connection to CNM license server ===
This step needs to be done prior to running COMSOL in your desktop session:
* Connect to <code>mega</code> with the following command in a terminal window:
ssh -L 1718:'''clicense1''':1718 -L 1719:'''clicense1''':1719 mega.cnm.anl.gov
* You can leave this connection running in your terminal window across several invocations of COMSOL.

=== Run COMSOL ===
# Ensure your connection to "mega" is still running. If not, restart it.
# Start the COMSOL application.

=== Troubleshooting ===
* If you get an error that the number of licenses is "exhausted", that is confirmation that you correctly reached the license server, though other users are using licenses.
* If the license access attempt takes a long time or hangs, the service might temporarily run on a secondary server. In this case, stop the mega session and restart it, connecting to the alternate license server:
ssh -L 1718:clicense'''2''':1718 -L 1719:clicense'''2''':1719 mega.cnm.anl.gov

== License Troubleshooting ==
The following are typical error messages.
=== Could not obtain license for COMSOL Multiphysics GUI ===
* Error details:
License error: -4.
'''Licensed number of users already reached.'''
Feature: COMSOLGUI
License path: /Users/stern/Applications/COMSOL52a/license/license.dat:
FlexNet Licensing error:-4,132
* Reason: The connection to the license server succeeded, but the license token(s) you requested are in use.
* Solution: Wait a bit, or [http://carbonmgmt/ganglia/lic.php identify who's using COMSOL] and politely ask the user when they might be finished with their work.

=== License server machine is down or not responding ===
* Error details:
License error: -96.
See the system administrator about starting the license server system, or
make sure you're referring to the right host (see LM_LICENSE_FILE).
Feature: SERIAL
Hostname: clicense3
License path: /Users/stern/Applications/COMSOL52a/license/license.dat:
FlexNet Licensing error:-96,7
* Reason: The license servers cannot be reached at all. The <code>ping</code> or <code>host</code> tests mentioned above fail.
* Solution: Review the client license file and DNS lookup domains.

=== License file does not support this version ===
* Error details:
Feature: SERIAL
'''Application version > License version: 5.21 > 3.5'''
License path: /Users/stern/Applications/COMSOL52a/license/license.dat:
FlexNet Licensing error:-21,126
* Reason: Only the first of two steps for connecting to the license servers succeeds, which may happen when the redundant license service runs on the alternate server instead of the one you're connected to.
* Solution: When using SSH, try <code>clicense2</code> instead of <code>clicense1</code>, or vice versa. Revisit your <code>~/.ssh/config</code> file or use the alternate command line as shown in the SSH section above.

== Multiple CPU cores ==
Ways to start multi-core (from http://www.comsol.com/support/knowledgebase/1096/)
comsol -np 8

export COMSOL_NUM_THREADS=8 # automatically provided on ''Carbon''
comsol
* COMSOL does not benefit from hyperthreading (oversubscribing the physical core count).

== Batch jobs ==

* http://www.comsol.com/support/knowledgebase/1001/ .
* [[Media:HPC 2012-05-16 Comsol parallel usage.pdf]] (Modified from http://www.comsol.com/support/knowledgebase/1001/files/4027/cluster_install_linux_42a.pptx – Appendix slide, PBS)
* '''Note:''' MPD is not supported on Carbon (I patched the main comsol script), and no longer necessary at all in Comsol-4.3.

=== Sample Job file ===
See:
$COMSOL_HOME/sample.job

== Important COMSOL Support Articles ==
* Toplevel suport page: https://www.comsol.com/support

=== KB Articles ===
* [https://www.comsol.com/support/knowledgebase/1001 Running COMSOL® in parallel on clusters]
* [https://www.comsol.com/support/knowledgebase/1250 Running parametric sweeps, batch sweeps, and cluster sweeps from the command line]
* [https://www.comsol.com/support/knowledgebase/1255 Reducing the amount of solution data stored in a model]
* [https://www.comsol.com/support/knowledgebase/830 Handling large models]

=== Blog-style articles ===
* [https://www.comsol.com/blogs/the-power-of-the-batch-sweep/ The Power of the Batch Sweep]
* [https://www.comsol.com/blogs/how-to-run-simulations-in-batch-mode-from-the-command-line/ How to Run Simulations in Batch Mode from the Command Line]

HPC/Applications/comsol

2026-02-26T17:14:48Z

Stern: /* Establish connection to CNM license server */

== Introduction ==
The [http://www.comsol.com/products/multiphysics/ '''COMSOL Multiphysics'''] application suite
is available for Mac, Windows, and Linux.

== Access ==
[[Image:HPC Comsol licensed modules.png|right|450px|thumb|COMSOL features licensed at CNM as of 2024-10.]]
=== COMSOL Multiphysics ===
Access to '''COMSOL Multiphysics''', the main application with both a graphical user interface and batch computing capabilities, will be granted '''''upon request''''' .

You must be:
* An '''''Argonne employee'''''  in one of the following roles:
** '''''NST staff member'''''  (including PostDocs, joint appointments, and students at NST).
** '''''CNM Facility User'''''  (including ''non-NST Argonne employees''), with an active proposal that has COMSOL in scope. Please see [[HPC/FAQ#Access_and_proposals]].

The set of available COMSOL modules that we have licensed is shown at right.

* The native desktop version can be installed only on ''Argonne-owned'' computers (laptop or desktop). Please contact [mailto:help@anl.gov?Subject=COMSOL%20license%20access%20request CNM support]. Make sure to review [http://www.comsol.com/support/updates/ Product Updates].
* To run COMSOL on our HPC system ''Carbon'', as GUI or as batch computing jobs, your proposal must have ''Carbon'' access in scope.

=== COMSOL Server ===
''CNM Facility Users'' who are not Argonne employees are not eligible to access
the full '''COMSOL Multiphysics''' product, in compliance with its licensing terms and conditions.
However, the [https://www.comsol.com/comsol-server '''COMSOL Server'''] product offers an alternative
albeit more narrow means of access.

All users, including '''''non-Argonne users''''' may be able to run, over a web interface in their browser,
pre-parameterized "fixed-model" simulations that were made with the '''COMSOL Application Builder''' (requires Windows).

To inquire about using COMSOL server applications,
please contact us.

We will review possible purchase paths for COMSOL Server, and the ''considerable effort'' likely required by users and the CNM to develop and deploy a COMSOL application.

== Support ==
* For questions on how to run or start COMSOL:
*: Submit an [[HPC/Support | '''IT support request''']]
* For questions related to most COMSOL data or scripts, contact COMSOL support at:
*: [mailto:support@comsol.com '''support'''@comsol.com]
* '''Exception:''' If your support question might contain data that is subject to '''US Export Control''' regulations, contact instead:
*: [mailto:us-export-control@us.comsol.com '''us-export-control'''@'''us'''.comsol.com]

== The modes to run COMSOL ==
There are several ways of running COMSOL.

=== Running the desktop application ===
To edit models and analyze results, use the COMSOL graphical user interface (GUI). You can run it in several ways, each explained in more detail in the following sections.

# Run on Carbon, display on your desktop using the [http://en.wikipedia.org/wiki/X_Window_System X Window system (X11)].
#* Requires an ''X11 Server'' application running on your client machine, and SSH X11 forwarding.
#* The X11 protocol can be slow over LAN.
# Run on Carbon within a [[HPC/Network Access/Virtual Desktop | virtual desktop environment (VNC)]].
#* Requires a VNC client on your machine.
#* It can be confusing where VNC and the app runs.
#* The virtual desktop presents a (deliberatly) sparse windowing environment – Copy&Paste can be difficult to use.
# Run on your Argonne-provided computer, checking out Carbon's license remotely.
#* Requires a full installation of COMSOL on your desktoo or laptop computer.
#* You will require access to the COMSOL license server running on Carbon, either through VPN (fairly easy) or SSH tunneling (can be intimidating for first-time users). See below for details.

=== Running a batch calculation ===
When the computational stage of models takes longer than a few minutes to complete, it will be beneficial to submit save the model file and run it in parallel on Carbon.
See the [[#Batch jobs|section below]].

== Use with X11 ==
For casual use, and with [[HPC/Network Access/SSH Tunnel Setup on Linux and MacOS | X11 tunneling to Carbon]] set up, simply add to your <code>~/.bashrc</code> file on Carbon:
module load comsol

Then, from the command line on a Carbon login node, start the COMSOL GUI:
comsol &

License configuration is automatic on Carbon.

Comsol will use a number of cores to run calculations multithreaded by default. (On Carbon, COMSOL_NUM_THREADS is set.)

'''Quit the application''' as soon as you are done. Only one user can use Comsol at a time.

== Use with VNC ==
See general [[HPC/Network Access/Virtual Desktop | instructions on using VNC]], and the COMSOL example therein.

== Native desktop version ==

To get the fastest response from the COMSOL GUI, run a native COMSOL version on your desktop machine or laptop, called ''client machine'' below.

Enter license details either in the COMSOL Setup configuration dialog.

== Native desktop version — License configuration ==
A client's COMSOL license needs to be configured depending on how the client reaches CNM network resources. Follow the appropriate section:

* [[#License configuration when inside CNM or on CNM VPN|License configuration when '''inside CNM''' or on '''CNM VPN''']]
: Requires that the user's machine is logically within the network of CNM/NST, either by a wired/auth WiFi connection in the building or by VPN under a CNM/NST profile.
* [[#License configuration when outside CNM – requires SSH Tunneling|License configuration when '''outside CNM''' – requires SSH Tunneling]]
: ''Non-NST staff'' usually have a VPN profile that places the machine logically [[#License configuration when outside CNM – requires SSH Tunneling|outside of NST's network.]]

== License configuration when inside CNM or on CNM VPN ==
=== Ensure correct DNS search domains ===
* Inspect the client machine's network configuration.
* Ensure that '''two''' entries are listed as DNS Search Domains for the current network profile:
nst.anl.gov
cnm.anl.gov
: (The order of entries does not matter for the purposes considered here.)
* If the client machine is ''wired to CNM networks'', the domains are normally provided correctly by DHCP.
* If the client machine is ''connected over VPN'', you may need edit the respective network profile.
:* On a Mac, find the VPN configuration details under System Preferences, Network, Advanced… . You may need to unlock the Network panel first before you can make changes under "Advanced…". It appears that those changes are being saved and will be activated automatically the next time that this VPN session gets started.
: [[Image:VNC_add_cnm_to_DNS_1.png|400px]] [[Image:VNC_add_cnm_to_DNS_2.png|400px]]
:* On Windows or Linux machines, take ''analogous'' steps, using the platform's own configuration tools.

* Exclude license manager components from client desktop installation. (Those are all handled server-side.)

=== License format ===
* Floating Network License (FNL), "Three-server redundancy"

=== License servers ===
* On client machines, while running the COMSOL installer or license configuration, specify license servers and ports as follows:
1718 clicense1
1718 clicense2
1718 clicense3
: '''Note:''' Use ''short host names only''. (Technical Reason: The license must be usable both on Carbon's internal network as well as on divisional networks).
1718 clicense1<strike>.cnm.anl.gov</strike>
1718 clicense2<strike>.cnm.anl.gov</strike>
1718 clicense3<strike>.cnm.anl.gov</strike>

'''Advanced:''' A client machine's COMSOL license file <code><COMSOLDIR>/Multiphysics/license/license.dat</code> should look as follows:
<source lang="make">
SERVER clicense1 ANY 1718
SERVER clicense2 ANY 1718
SERVER clicense3 ANY 1718
USE_SERVER
</source>

<code><COMSOLDIR></code> is located by default at:
* '''Windows''': C:\Program files\COMSOL\COMSOL''nn''
* '''Linux''': /usr/local/comsol''nn''
* '''Mac''': /Applications/COMSOL''nn''
("''nn''" is the application's abbreviated version number.)

=== Network Troubleshooting ===
When using the license access model descibed here (as opposed to tunneled over SSH), ensure the following:
* Verify that short host names can be resolved to IP addresses. Either one of the following commands should produce sensible output:
host clicense1
: or:
ping -c 3 clicense1
* Verify that ''both license ports on the same server'' can be reached from your client machine:
telnet '''clicense1''' 1718
telnet '''clicense1''' 1719
: or:
telnet '''clicense2''' 1718
telnet '''clicense2''' 1719
: A ''successful'' connection looks like:
Trying 146.139.x.y...
Connected to clicense1.cnm.anl.gov.
Escape character is '^]'.
:: Do not enter any data. Instead, close each connection by hitting <code>Ctrl</code><code>]</code> (closing square bracket), followed by <code>Ctrl</code><code>D</code>.
: A ''failed'' connection gives one of these errors:
telnet: connect to address 146.139.x.y: Operation timed out
: or:
telnet: connect to address 146.139.x.y: Connection refused
:: In these cases, ask that your IP firewall configuration be reviewed and changed.

== License configuration when outside CNM – requires SSH Tunneling ==
This section documents how to run COMSOL from a network location outside CNM.

As a one-time preparation step, two items must be configured:
* The license host that COMSOL is to contact, and
* An SSH connection that will allow your machine to access the COMSOL license server inside CNM's network.

To then ''run'' the COMSOL application, the configured SSH connection must be up and kept running for the entire time that you run the COMSOL. When you exit COMSOL and wish to start it again, you can keep the SSH connection running, it will be re-used for the next COMSOL run.

=== Configure COMSOL ===
The following needs to be done just once, typically during COMSOL installation:

In the COMSOL Installer, or in COMSOL Preferences, enter:
* '''License format:''' Floating Network License (FNL), "Port number and Hostname"
:: Do ''not use'' "Three-server redundancy", since it cannot (easily) be accessed over SSH. Configure with just one server, and in the rare situation of a server failover, temporarily tweak your SSH configuration to contact the alternate server (see below).
* '''Server:''' <code>127.0.0.1</code>
:: Do not use "localhost", as COMSOL interprets this incorrectly to mean that your machine ''is'' a license server.

* '''Port:''' <code>1718</code>


=== Establish connection to CNM license server ===
This step needs to be done prior to running COMSOL in your desktop session:
* Connect to <code>mega</code> with the following ssh command:
ssh -L 1718:'''clicense1''':1718 -L 1719:'''clicense1''':1719 mega.cnm.anl.gov
* Leave the command running between invocations of COMSOL.

=== Run COMSOL ===
# Ensure your connection to "mega" is still running. If not, restart it.
# Start the COMSOL application.

=== Troubleshooting ===
* If you get an error that the number of licenses is "exhausted", that is confirmation that you correctly reached the license server, though other users are using licenses.
* If the license access attempt takes a long time or hangs, the service might temporarily run on a secondary server. In this case, stop the mega session and restart it, connecting to the alternate license server:
ssh -L 1718:clicense'''2''':1718 -L 1719:clicense'''2''':1719 mega.cnm.anl.gov

== License Troubleshooting ==
The following are typical error messages.
=== Could not obtain license for COMSOL Multiphysics GUI ===
* Error details:
License error: -4.
'''Licensed number of users already reached.'''
Feature: COMSOLGUI
License path: /Users/stern/Applications/COMSOL52a/license/license.dat:
FlexNet Licensing error:-4,132
* Reason: The connection to the license server succeeded, but the license token(s) you requested are in use.
* Solution: Wait a bit, or [http://carbonmgmt/ganglia/lic.php identify who's using COMSOL] and politely ask the user when they might be finished with their work.

=== License server machine is down or not responding ===
* Error details:
License error: -96.
See the system administrator about starting the license server system, or
make sure you're referring to the right host (see LM_LICENSE_FILE).
Feature: SERIAL
Hostname: clicense3
License path: /Users/stern/Applications/COMSOL52a/license/license.dat:
FlexNet Licensing error:-96,7
* Reason: The license servers cannot be reached at all. The <code>ping</code> or <code>host</code> tests mentioned above fail.
* Solution: Review the client license file and DNS lookup domains.

=== License file does not support this version ===
* Error details:
Feature: SERIAL
'''Application version > License version: 5.21 > 3.5'''
License path: /Users/stern/Applications/COMSOL52a/license/license.dat:
FlexNet Licensing error:-21,126
* Reason: Only the first of two steps for connecting to the license servers succeeds, which may happen when the redundant license service runs on the alternate server instead of the one you're connected to.
* Solution: When using SSH, try <code>clicense2</code> instead of <code>clicense1</code>, or vice versa. Revisit your <code>~/.ssh/config</code> file or use the alternate command line as shown in the SSH section above.

== Multiple CPU cores ==
Ways to start multi-core (from http://www.comsol.com/support/knowledgebase/1096/)
comsol -np 8

export COMSOL_NUM_THREADS=8 # automatically provided on ''Carbon''
comsol
* COMSOL does not benefit from hyperthreading (oversubscribing the physical core count).

== Batch jobs ==

* http://www.comsol.com/support/knowledgebase/1001/ .
* [[Media:HPC 2012-05-16 Comsol parallel usage.pdf]] (Modified from http://www.comsol.com/support/knowledgebase/1001/files/4027/cluster_install_linux_42a.pptx – Appendix slide, PBS)
* '''Note:''' MPD is not supported on Carbon (I patched the main comsol script), and no longer necessary at all in Comsol-4.3.

=== Sample Job file ===
See:
$COMSOL_HOME/sample.job

== Important COMSOL Support Articles ==
* Toplevel suport page: https://www.comsol.com/support

=== KB Articles ===
* [https://www.comsol.com/support/knowledgebase/1001 Running COMSOL® in parallel on clusters]
* [https://www.comsol.com/support/knowledgebase/1250 Running parametric sweeps, batch sweeps, and cluster sweeps from the command line]
* [https://www.comsol.com/support/knowledgebase/1255 Reducing the amount of solution data stored in a model]
* [https://www.comsol.com/support/knowledgebase/830 Handling large models]

=== Blog-style articles ===
* [https://www.comsol.com/blogs/the-power-of-the-batch-sweep/ The Power of the Batch Sweep]
* [https://www.comsol.com/blogs/how-to-run-simulations-in-batch-mode-from-the-command-line/ How to Run Simulations in Batch Mode from the Command Line]

HPC/Applications/comsol

2026-02-26T17:14:07Z

Stern: /* Configure COMSOL */

== Introduction ==
The [http://www.comsol.com/products/multiphysics/ '''COMSOL Multiphysics'''] application suite
is available for Mac, Windows, and Linux.

== Access ==
[[Image:HPC Comsol licensed modules.png|right|450px|thumb|COMSOL features licensed at CNM as of 2024-10.]]
=== COMSOL Multiphysics ===
Access to '''COMSOL Multiphysics''', the main application with both a graphical user interface and batch computing capabilities, will be granted '''''upon request''''' .

You must be:
* An '''''Argonne employee'''''  in one of the following roles:
** '''''NST staff member'''''  (including PostDocs, joint appointments, and students at NST).
** '''''CNM Facility User'''''  (including ''non-NST Argonne employees''), with an active proposal that has COMSOL in scope. Please see [[HPC/FAQ#Access_and_proposals]].

The set of available COMSOL modules that we have licensed is shown at right.

* The native desktop version can be installed only on ''Argonne-owned'' computers (laptop or desktop). Please contact [mailto:help@anl.gov?Subject=COMSOL%20license%20access%20request CNM support]. Make sure to review [http://www.comsol.com/support/updates/ Product Updates].
* To run COMSOL on our HPC system ''Carbon'', as GUI or as batch computing jobs, your proposal must have ''Carbon'' access in scope.

=== COMSOL Server ===
''CNM Facility Users'' who are not Argonne employees are not eligible to access
the full '''COMSOL Multiphysics''' product, in compliance with its licensing terms and conditions.
However, the [https://www.comsol.com/comsol-server '''COMSOL Server'''] product offers an alternative
albeit more narrow means of access.

All users, including '''''non-Argonne users''''' may be able to run, over a web interface in their browser,
pre-parameterized "fixed-model" simulations that were made with the '''COMSOL Application Builder''' (requires Windows).

To inquire about using COMSOL server applications,
please contact us.

We will review possible purchase paths for COMSOL Server, and the ''considerable effort'' likely required by users and the CNM to develop and deploy a COMSOL application.

== Support ==
* For questions on how to run or start COMSOL:
*: Submit an [[HPC/Support | '''IT support request''']]
* For questions related to most COMSOL data or scripts, contact COMSOL support at:
*: [mailto:support@comsol.com '''support'''@comsol.com]
* '''Exception:''' If your support question might contain data that is subject to '''US Export Control''' regulations, contact instead:
*: [mailto:us-export-control@us.comsol.com '''us-export-control'''@'''us'''.comsol.com]

== The modes to run COMSOL ==
There are several ways of running COMSOL.

=== Running the desktop application ===
To edit models and analyze results, use the COMSOL graphical user interface (GUI). You can run it in several ways, each explained in more detail in the following sections.

# Run on Carbon, display on your desktop using the [http://en.wikipedia.org/wiki/X_Window_System X Window system (X11)].
#* Requires an ''X11 Server'' application running on your client machine, and SSH X11 forwarding.
#* The X11 protocol can be slow over LAN.
# Run on Carbon within a [[HPC/Network Access/Virtual Desktop | virtual desktop environment (VNC)]].
#* Requires a VNC client on your machine.
#* It can be confusing where VNC and the app runs.
#* The virtual desktop presents a (deliberatly) sparse windowing environment – Copy&Paste can be difficult to use.
# Run on your Argonne-provided computer, checking out Carbon's license remotely.
#* Requires a full installation of COMSOL on your desktoo or laptop computer.
#* You will require access to the COMSOL license server running on Carbon, either through VPN (fairly easy) or SSH tunneling (can be intimidating for first-time users). See below for details.

=== Running a batch calculation ===
When the computational stage of models takes longer than a few minutes to complete, it will be beneficial to submit save the model file and run it in parallel on Carbon.
See the [[#Batch jobs|section below]].

== Use with X11 ==
For casual use, and with [[HPC/Network Access/SSH Tunnel Setup on Linux and MacOS | X11 tunneling to Carbon]] set up, simply add to your <code>~/.bashrc</code> file on Carbon:
module load comsol

Then, from the command line on a Carbon login node, start the COMSOL GUI:
comsol &

License configuration is automatic on Carbon.

Comsol will use a number of cores to run calculations multithreaded by default. (On Carbon, COMSOL_NUM_THREADS is set.)

'''Quit the application''' as soon as you are done. Only one user can use Comsol at a time.

== Use with VNC ==
See general [[HPC/Network Access/Virtual Desktop | instructions on using VNC]], and the COMSOL example therein.

== Native desktop version ==

To get the fastest response from the COMSOL GUI, run a native COMSOL version on your desktop machine or laptop, called ''client machine'' below.

Enter license details either in the COMSOL Setup configuration dialog.

== Native desktop version — License configuration ==
A client's COMSOL license needs to be configured depending on how the client reaches CNM network resources. Follow the appropriate section:

* [[#License configuration when inside CNM or on CNM VPN|License configuration when '''inside CNM''' or on '''CNM VPN''']]
: Requires that the user's machine is logically within the network of CNM/NST, either by a wired/auth WiFi connection in the building or by VPN under a CNM/NST profile.
* [[#License configuration when outside CNM – requires SSH Tunneling|License configuration when '''outside CNM''' – requires SSH Tunneling]]
: ''Non-NST staff'' usually have a VPN profile that places the machine logically [[#License configuration when outside CNM – requires SSH Tunneling|outside of NST's network.]]

== License configuration when inside CNM or on CNM VPN ==
=== Ensure correct DNS search domains ===
* Inspect the client machine's network configuration.
* Ensure that '''two''' entries are listed as DNS Search Domains for the current network profile:
nst.anl.gov
cnm.anl.gov
: (The order of entries does not matter for the purposes considered here.)
* If the client machine is ''wired to CNM networks'', the domains are normally provided correctly by DHCP.
* If the client machine is ''connected over VPN'', you may need edit the respective network profile.
:* On a Mac, find the VPN configuration details under System Preferences, Network, Advanced… . You may need to unlock the Network panel first before you can make changes under "Advanced…". It appears that those changes are being saved and will be activated automatically the next time that this VPN session gets started.
: [[Image:VNC_add_cnm_to_DNS_1.png|400px]] [[Image:VNC_add_cnm_to_DNS_2.png|400px]]
:* On Windows or Linux machines, take ''analogous'' steps, using the platform's own configuration tools.

* Exclude license manager components from client desktop installation. (Those are all handled server-side.)

=== License format ===
* Floating Network License (FNL), "Three-server redundancy"

=== License servers ===
* On client machines, while running the COMSOL installer or license configuration, specify license servers and ports as follows:
1718 clicense1
1718 clicense2
1718 clicense3
: '''Note:''' Use ''short host names only''. (Technical Reason: The license must be usable both on Carbon's internal network as well as on divisional networks).
1718 clicense1<strike>.cnm.anl.gov</strike>
1718 clicense2<strike>.cnm.anl.gov</strike>
1718 clicense3<strike>.cnm.anl.gov</strike>

'''Advanced:''' A client machine's COMSOL license file <code><COMSOLDIR>/Multiphysics/license/license.dat</code> should look as follows:
<source lang="make">
SERVER clicense1 ANY 1718
SERVER clicense2 ANY 1718
SERVER clicense3 ANY 1718
USE_SERVER
</source>

<code><COMSOLDIR></code> is located by default at:
* '''Windows''': C:\Program files\COMSOL\COMSOL''nn''
* '''Linux''': /usr/local/comsol''nn''
* '''Mac''': /Applications/COMSOL''nn''
("''nn''" is the application's abbreviated version number.)

=== Network Troubleshooting ===
When using the license access model descibed here (as opposed to tunneled over SSH), ensure the following:
* Verify that short host names can be resolved to IP addresses. Either one of the following commands should produce sensible output:
host clicense1
: or:
ping -c 3 clicense1
* Verify that ''both license ports on the same server'' can be reached from your client machine:
telnet '''clicense1''' 1718
telnet '''clicense1''' 1719
: or:
telnet '''clicense2''' 1718
telnet '''clicense2''' 1719
: A ''successful'' connection looks like:
Trying 146.139.x.y...
Connected to clicense1.cnm.anl.gov.
Escape character is '^]'.
:: Do not enter any data. Instead, close each connection by hitting <code>Ctrl</code><code>]</code> (closing square bracket), followed by <code>Ctrl</code><code>D</code>.
: A ''failed'' connection gives one of these errors:
telnet: connect to address 146.139.x.y: Operation timed out
: or:
telnet: connect to address 146.139.x.y: Connection refused
:: In these cases, ask that your IP firewall configuration be reviewed and changed.

== License configuration when outside CNM – requires SSH Tunneling ==
This section documents how to run COMSOL from a network location outside CNM.

As a one-time preparation step, two items must be configured:
* The license host that COMSOL is to contact, and
* An SSH connection that will allow your machine to access the COMSOL license server inside CNM's network.

To then ''run'' the COMSOL application, the configured SSH connection must be up and kept running for the entire time that you run the COMSOL. When you exit COMSOL and wish to start it again, you can keep the SSH connection running, it will be re-used for the next COMSOL run.

=== Configure COMSOL ===
The following needs to be done just once, typically during COMSOL installation:

In the COMSOL Installer, or in COMSOL Preferences, enter:
* '''License format:''' Floating Network License (FNL), "Port number and Hostname"
:: Do ''not use'' "Three-server redundancy", since it cannot (easily) be accessed over SSH. Configure with just one server, and in the rare situation of a server failover, temporarily tweak your SSH configuration to contact the alternate server (see below).
* '''Server:''' <code>127.0.0.1</code>
:: Do not use "localhost", as COMSOL interprets this incorrectly to mean that your machine ''is'' a license server.

* '''Port:''' <code>1718</code>


=== Establish connection to CNM license server ===
This step needs to be done prior to running COMSOL in your desktop session:
# Connect to <code>mega</code> with the following ssh command:
ssh -L 1718:'''clicense1''':1718 -L 1719:'''clicense1''':1719 mega.cnm.anl.gov
# Leave the command running between invocations of COMSOL.

=== Run COMSOL ===
# Ensure your connection to "mega" is still running. If not, restart it.
# Start the COMSOL application.

=== Troubleshooting ===
* If you get an error that the number of licenses is "exhausted", that is confirmation that you correctly reached the license server, though other users are using licenses.
* If the license access attempt takes a long time or hangs, the service might temporarily run on a secondary server. In this case, stop the mega session and restart it, connecting to the alternate license server:
ssh -L 1718:clicense'''2''':1718 -L 1719:clicense'''2''':1719 mega.cnm.anl.gov

== License Troubleshooting ==
The following are typical error messages.
=== Could not obtain license for COMSOL Multiphysics GUI ===
* Error details:
License error: -4.
'''Licensed number of users already reached.'''
Feature: COMSOLGUI
License path: /Users/stern/Applications/COMSOL52a/license/license.dat:
FlexNet Licensing error:-4,132
* Reason: The connection to the license server succeeded, but the license token(s) you requested are in use.
* Solution: Wait a bit, or [http://carbonmgmt/ganglia/lic.php identify who's using COMSOL] and politely ask the user when they might be finished with their work.

=== License server machine is down or not responding ===
* Error details:
License error: -96.
See the system administrator about starting the license server system, or
make sure you're referring to the right host (see LM_LICENSE_FILE).
Feature: SERIAL
Hostname: clicense3
License path: /Users/stern/Applications/COMSOL52a/license/license.dat:
FlexNet Licensing error:-96,7
* Reason: The license servers cannot be reached at all. The <code>ping</code> or <code>host</code> tests mentioned above fail.
* Solution: Review the client license file and DNS lookup domains.

=== License file does not support this version ===
* Error details:
Feature: SERIAL
'''Application version > License version: 5.21 > 3.5'''
License path: /Users/stern/Applications/COMSOL52a/license/license.dat:
FlexNet Licensing error:-21,126
* Reason: Only the first of two steps for connecting to the license servers succeeds, which may happen when the redundant license service runs on the alternate server instead of the one you're connected to.
* Solution: When using SSH, try <code>clicense2</code> instead of <code>clicense1</code>, or vice versa. Revisit your <code>~/.ssh/config</code> file or use the alternate command line as shown in the SSH section above.

== Multiple CPU cores ==
Ways to start multi-core (from http://www.comsol.com/support/knowledgebase/1096/)
comsol -np 8

export COMSOL_NUM_THREADS=8 # automatically provided on ''Carbon''
comsol
* COMSOL does not benefit from hyperthreading (oversubscribing the physical core count).

== Batch jobs ==

* http://www.comsol.com/support/knowledgebase/1001/ .
* [[Media:HPC 2012-05-16 Comsol parallel usage.pdf]] (Modified from http://www.comsol.com/support/knowledgebase/1001/files/4027/cluster_install_linux_42a.pptx – Appendix slide, PBS)
* '''Note:''' MPD is not supported on Carbon (I patched the main comsol script), and no longer necessary at all in Comsol-4.3.

=== Sample Job file ===
See:
$COMSOL_HOME/sample.job

== Important COMSOL Support Articles ==
* Toplevel suport page: https://www.comsol.com/support

=== KB Articles ===
* [https://www.comsol.com/support/knowledgebase/1001 Running COMSOL® in parallel on clusters]
* [https://www.comsol.com/support/knowledgebase/1250 Running parametric sweeps, batch sweeps, and cluster sweeps from the command line]
* [https://www.comsol.com/support/knowledgebase/1255 Reducing the amount of solution data stored in a model]
* [https://www.comsol.com/support/knowledgebase/830 Handling large models]

=== Blog-style articles ===
* [https://www.comsol.com/blogs/the-power-of-the-batch-sweep/ The Power of the Batch Sweep]
* [https://www.comsol.com/blogs/how-to-run-simulations-in-batch-mode-from-the-command-line/ How to Run Simulations in Batch Mode from the Command Line]

HPC/Applications/comsol

2026-02-26T17:13:49Z

Stern: /* Configure COMSOL */

== Introduction ==
The [http://www.comsol.com/products/multiphysics/ '''COMSOL Multiphysics'''] application suite
is available for Mac, Windows, and Linux.

== Access ==
[[Image:HPC Comsol licensed modules.png|right|450px|thumb|COMSOL features licensed at CNM as of 2024-10.]]
=== COMSOL Multiphysics ===
Access to '''COMSOL Multiphysics''', the main application with both a graphical user interface and batch computing capabilities, will be granted '''''upon request''''' .

You must be:
* An '''''Argonne employee'''''  in one of the following roles:
** '''''NST staff member'''''  (including PostDocs, joint appointments, and students at NST).
** '''''CNM Facility User'''''  (including ''non-NST Argonne employees''), with an active proposal that has COMSOL in scope. Please see [[HPC/FAQ#Access_and_proposals]].

The set of available COMSOL modules that we have licensed is shown at right.

* The native desktop version can be installed only on ''Argonne-owned'' computers (laptop or desktop). Please contact [mailto:help@anl.gov?Subject=COMSOL%20license%20access%20request CNM support]. Make sure to review [http://www.comsol.com/support/updates/ Product Updates].
* To run COMSOL on our HPC system ''Carbon'', as GUI or as batch computing jobs, your proposal must have ''Carbon'' access in scope.

=== COMSOL Server ===
''CNM Facility Users'' who are not Argonne employees are not eligible to access
the full '''COMSOL Multiphysics''' product, in compliance with its licensing terms and conditions.
However, the [https://www.comsol.com/comsol-server '''COMSOL Server'''] product offers an alternative
albeit more narrow means of access.

All users, including '''''non-Argonne users''''' may be able to run, over a web interface in their browser,
pre-parameterized "fixed-model" simulations that were made with the '''COMSOL Application Builder''' (requires Windows).

To inquire about using COMSOL server applications,
please contact us.

We will review possible purchase paths for COMSOL Server, and the ''considerable effort'' likely required by users and the CNM to develop and deploy a COMSOL application.

== Support ==
* For questions on how to run or start COMSOL:
*: Submit an [[HPC/Support | '''IT support request''']]
* For questions related to most COMSOL data or scripts, contact COMSOL support at:
*: [mailto:support@comsol.com '''support'''@comsol.com]
* '''Exception:''' If your support question might contain data that is subject to '''US Export Control''' regulations, contact instead:
*: [mailto:us-export-control@us.comsol.com '''us-export-control'''@'''us'''.comsol.com]

== The modes to run COMSOL ==
There are several ways of running COMSOL.

=== Running the desktop application ===
To edit models and analyze results, use the COMSOL graphical user interface (GUI). You can run it in several ways, each explained in more detail in the following sections.

# Run on Carbon, display on your desktop using the [http://en.wikipedia.org/wiki/X_Window_System X Window system (X11)].
#* Requires an ''X11 Server'' application running on your client machine, and SSH X11 forwarding.
#* The X11 protocol can be slow over LAN.
# Run on Carbon within a [[HPC/Network Access/Virtual Desktop | virtual desktop environment (VNC)]].
#* Requires a VNC client on your machine.
#* It can be confusing where VNC and the app runs.
#* The virtual desktop presents a (deliberatly) sparse windowing environment – Copy&Paste can be difficult to use.
# Run on your Argonne-provided computer, checking out Carbon's license remotely.
#* Requires a full installation of COMSOL on your desktoo or laptop computer.
#* You will require access to the COMSOL license server running on Carbon, either through VPN (fairly easy) or SSH tunneling (can be intimidating for first-time users). See below for details.

=== Running a batch calculation ===
When the computational stage of models takes longer than a few minutes to complete, it will be beneficial to submit save the model file and run it in parallel on Carbon.
See the [[#Batch jobs|section below]].

== Use with X11 ==
For casual use, and with [[HPC/Network Access/SSH Tunnel Setup on Linux and MacOS | X11 tunneling to Carbon]] set up, simply add to your <code>~/.bashrc</code> file on Carbon:
module load comsol

Then, from the command line on a Carbon login node, start the COMSOL GUI:
comsol &

License configuration is automatic on Carbon.

Comsol will use a number of cores to run calculations multithreaded by default. (On Carbon, COMSOL_NUM_THREADS is set.)

'''Quit the application''' as soon as you are done. Only one user can use Comsol at a time.

== Use with VNC ==
See general [[HPC/Network Access/Virtual Desktop | instructions on using VNC]], and the COMSOL example therein.

== Native desktop version ==

To get the fastest response from the COMSOL GUI, run a native COMSOL version on your desktop machine or laptop, called ''client machine'' below.

Enter license details either in the COMSOL Setup configuration dialog.

== Native desktop version — License configuration ==
A client's COMSOL license needs to be configured depending on how the client reaches CNM network resources. Follow the appropriate section:

* [[#License configuration when inside CNM or on CNM VPN|License configuration when '''inside CNM''' or on '''CNM VPN''']]
: Requires that the user's machine is logically within the network of CNM/NST, either by a wired/auth WiFi connection in the building or by VPN under a CNM/NST profile.
* [[#License configuration when outside CNM – requires SSH Tunneling|License configuration when '''outside CNM''' – requires SSH Tunneling]]
: ''Non-NST staff'' usually have a VPN profile that places the machine logically [[#License configuration when outside CNM – requires SSH Tunneling|outside of NST's network.]]

== License configuration when inside CNM or on CNM VPN ==
=== Ensure correct DNS search domains ===
* Inspect the client machine's network configuration.
* Ensure that '''two''' entries are listed as DNS Search Domains for the current network profile:
nst.anl.gov
cnm.anl.gov
: (The order of entries does not matter for the purposes considered here.)
* If the client machine is ''wired to CNM networks'', the domains are normally provided correctly by DHCP.
* If the client machine is ''connected over VPN'', you may need edit the respective network profile.
:* On a Mac, find the VPN configuration details under System Preferences, Network, Advanced… . You may need to unlock the Network panel first before you can make changes under "Advanced…". It appears that those changes are being saved and will be activated automatically the next time that this VPN session gets started.
: [[Image:VNC_add_cnm_to_DNS_1.png|400px]] [[Image:VNC_add_cnm_to_DNS_2.png|400px]]
:* On Windows or Linux machines, take ''analogous'' steps, using the platform's own configuration tools.

* Exclude license manager components from client desktop installation. (Those are all handled server-side.)

=== License format ===
* Floating Network License (FNL), "Three-server redundancy"

=== License servers ===
* On client machines, while running the COMSOL installer or license configuration, specify license servers and ports as follows:
1718 clicense1
1718 clicense2
1718 clicense3
: '''Note:''' Use ''short host names only''. (Technical Reason: The license must be usable both on Carbon's internal network as well as on divisional networks).
1718 clicense1<strike>.cnm.anl.gov</strike>
1718 clicense2<strike>.cnm.anl.gov</strike>
1718 clicense3<strike>.cnm.anl.gov</strike>

'''Advanced:''' A client machine's COMSOL license file <code><COMSOLDIR>/Multiphysics/license/license.dat</code> should look as follows:
<source lang="make">
SERVER clicense1 ANY 1718
SERVER clicense2 ANY 1718
SERVER clicense3 ANY 1718
USE_SERVER
</source>

<code><COMSOLDIR></code> is located by default at:
* '''Windows''': C:\Program files\COMSOL\COMSOL''nn''
* '''Linux''': /usr/local/comsol''nn''
* '''Mac''': /Applications/COMSOL''nn''
("''nn''" is the application's abbreviated version number.)

=== Network Troubleshooting ===
When using the license access model descibed here (as opposed to tunneled over SSH), ensure the following:
* Verify that short host names can be resolved to IP addresses. Either one of the following commands should produce sensible output:
host clicense1
: or:
ping -c 3 clicense1
* Verify that ''both license ports on the same server'' can be reached from your client machine:
telnet '''clicense1''' 1718
telnet '''clicense1''' 1719
: or:
telnet '''clicense2''' 1718
telnet '''clicense2''' 1719
: A ''successful'' connection looks like:
Trying 146.139.x.y...
Connected to clicense1.cnm.anl.gov.
Escape character is '^]'.
:: Do not enter any data. Instead, close each connection by hitting <code>Ctrl</code><code>]</code> (closing square bracket), followed by <code>Ctrl</code><code>D</code>.
: A ''failed'' connection gives one of these errors:
telnet: connect to address 146.139.x.y: Operation timed out
: or:
telnet: connect to address 146.139.x.y: Connection refused
:: In these cases, ask that your IP firewall configuration be reviewed and changed.

== License configuration when outside CNM – requires SSH Tunneling ==
This section documents how to run COMSOL from a network location outside CNM.

As a one-time preparation step, two items must be configured:
* The license host that COMSOL is to contact, and
* An SSH connection that will allow your machine to access the COMSOL license server inside CNM's network.

To then ''run'' the COMSOL application, the configured SSH connection must be up and kept running for the entire time that you run the COMSOL. When you exit COMSOL and wish to start it again, you can keep the SSH connection running, it will be re-used for the next COMSOL run.

=== Configure COMSOL ===
The following needs to be done just once, typically during COMSOL installation:

In the COMSOL Installer, or in COMSOL Preferences, enter:
* '''License format:''' Floating Network License (FNL), "Port number and Hostname"
:: Do ''not use'' "Three-server redundancy", since it cannot (easily) be accessed over SSH. Configure with just one server, and in the rare situation of a server failover, tweak your SSH configuration to contact the alternate server.
* '''Server:''' <code>127.0.0.1</code>
:: Do not use "localhost", as COMSOL interprets this incorrectly to mean that your machine ''is'' a license server.

* '''Port:''' <code>1718</code>


=== Establish connection to CNM license server ===
This step needs to be done prior to running COMSOL in your desktop session:
# Connect to <code>mega</code> with the following ssh command:
ssh -L 1718:'''clicense1''':1718 -L 1719:'''clicense1''':1719 mega.cnm.anl.gov
# Leave the command running between invocations of COMSOL.

=== Run COMSOL ===
# Ensure your connection to "mega" is still running. If not, restart it.
# Start the COMSOL application.

=== Troubleshooting ===
* If you get an error that the number of licenses is "exhausted", that is confirmation that you correctly reached the license server, though other users are using licenses.
* If the license access attempt takes a long time or hangs, the service might temporarily run on a secondary server. In this case, stop the mega session and restart it, connecting to the alternate license server:
ssh -L 1718:clicense'''2''':1718 -L 1719:clicense'''2''':1719 mega.cnm.anl.gov

== License Troubleshooting ==
The following are typical error messages.
=== Could not obtain license for COMSOL Multiphysics GUI ===
* Error details:
License error: -4.
'''Licensed number of users already reached.'''
Feature: COMSOLGUI
License path: /Users/stern/Applications/COMSOL52a/license/license.dat:
FlexNet Licensing error:-4,132
* Reason: The connection to the license server succeeded, but the license token(s) you requested are in use.
* Solution: Wait a bit, or [http://carbonmgmt/ganglia/lic.php identify who's using COMSOL] and politely ask the user when they might be finished with their work.

=== License server machine is down or not responding ===
* Error details:
License error: -96.
See the system administrator about starting the license server system, or
make sure you're referring to the right host (see LM_LICENSE_FILE).
Feature: SERIAL
Hostname: clicense3
License path: /Users/stern/Applications/COMSOL52a/license/license.dat:
FlexNet Licensing error:-96,7
* Reason: The license servers cannot be reached at all. The <code>ping</code> or <code>host</code> tests mentioned above fail.
* Solution: Review the client license file and DNS lookup domains.

=== License file does not support this version ===
* Error details:
Feature: SERIAL
'''Application version > License version: 5.21 > 3.5'''
License path: /Users/stern/Applications/COMSOL52a/license/license.dat:
FlexNet Licensing error:-21,126
* Reason: Only the first of two steps for connecting to the license servers succeeds, which may happen when the redundant license service runs on the alternate server instead of the one you're connected to.
* Solution: When using SSH, try <code>clicense2</code> instead of <code>clicense1</code>, or vice versa. Revisit your <code>~/.ssh/config</code> file or use the alternate command line as shown in the SSH section above.

== Multiple CPU cores ==
Ways to start multi-core (from http://www.comsol.com/support/knowledgebase/1096/)
comsol -np 8

export COMSOL_NUM_THREADS=8 # automatically provided on ''Carbon''
comsol
* COMSOL does not benefit from hyperthreading (oversubscribing the physical core count).

== Batch jobs ==

* http://www.comsol.com/support/knowledgebase/1001/ .
* [[Media:HPC 2012-05-16 Comsol parallel usage.pdf]] (Modified from http://www.comsol.com/support/knowledgebase/1001/files/4027/cluster_install_linux_42a.pptx – Appendix slide, PBS)
* '''Note:''' MPD is not supported on Carbon (I patched the main comsol script), and no longer necessary at all in Comsol-4.3.

=== Sample Job file ===
See:
$COMSOL_HOME/sample.job

== Important COMSOL Support Articles ==
* Toplevel suport page: https://www.comsol.com/support

=== KB Articles ===
* [https://www.comsol.com/support/knowledgebase/1001 Running COMSOL® in parallel on clusters]
* [https://www.comsol.com/support/knowledgebase/1250 Running parametric sweeps, batch sweeps, and cluster sweeps from the command line]
* [https://www.comsol.com/support/knowledgebase/1255 Reducing the amount of solution data stored in a model]
* [https://www.comsol.com/support/knowledgebase/830 Handling large models]

=== Blog-style articles ===
* [https://www.comsol.com/blogs/the-power-of-the-batch-sweep/ The Power of the Batch Sweep]
* [https://www.comsol.com/blogs/how-to-run-simulations-in-batch-mode-from-the-command-line/ How to Run Simulations in Batch Mode from the Command Line]

HPC/Applications/comsol

2026-02-26T17:11:15Z

Stern: /* License configuration when outside CNM – requires SSH Tunneling */

== Introduction ==
The [http://www.comsol.com/products/multiphysics/ '''COMSOL Multiphysics'''] application suite
is available for Mac, Windows, and Linux.

== Access ==
[[Image:HPC Comsol licensed modules.png|right|450px|thumb|COMSOL features licensed at CNM as of 2024-10.]]
=== COMSOL Multiphysics ===
Access to '''COMSOL Multiphysics''', the main application with both a graphical user interface and batch computing capabilities, will be granted '''''upon request''''' .

You must be:
* An '''''Argonne employee'''''  in one of the following roles:
** '''''NST staff member'''''  (including PostDocs, joint appointments, and students at NST).
** '''''CNM Facility User'''''  (including ''non-NST Argonne employees''), with an active proposal that has COMSOL in scope. Please see [[HPC/FAQ#Access_and_proposals]].

The set of available COMSOL modules that we have licensed is shown at right.

* The native desktop version can be installed only on ''Argonne-owned'' computers (laptop or desktop). Please contact [mailto:help@anl.gov?Subject=COMSOL%20license%20access%20request CNM support]. Make sure to review [http://www.comsol.com/support/updates/ Product Updates].
* To run COMSOL on our HPC system ''Carbon'', as GUI or as batch computing jobs, your proposal must have ''Carbon'' access in scope.

=== COMSOL Server ===
''CNM Facility Users'' who are not Argonne employees are not eligible to access
the full '''COMSOL Multiphysics''' product, in compliance with its licensing terms and conditions.
However, the [https://www.comsol.com/comsol-server '''COMSOL Server'''] product offers an alternative
albeit more narrow means of access.

All users, including '''''non-Argonne users''''' may be able to run, over a web interface in their browser,
pre-parameterized "fixed-model" simulations that were made with the '''COMSOL Application Builder''' (requires Windows).

To inquire about using COMSOL server applications,
please contact us.

We will review possible purchase paths for COMSOL Server, and the ''considerable effort'' likely required by users and the CNM to develop and deploy a COMSOL application.

== Support ==
* For questions on how to run or start COMSOL:
*: Submit an [[HPC/Support | '''IT support request''']]
* For questions related to most COMSOL data or scripts, contact COMSOL support at:
*: [mailto:support@comsol.com '''support'''@comsol.com]
* '''Exception:''' If your support question might contain data that is subject to '''US Export Control''' regulations, contact instead:
*: [mailto:us-export-control@us.comsol.com '''us-export-control'''@'''us'''.comsol.com]

== The modes to run COMSOL ==
There are several ways of running COMSOL.

=== Running the desktop application ===
To edit models and analyze results, use the COMSOL graphical user interface (GUI). You can run it in several ways, each explained in more detail in the following sections.

# Run on Carbon, display on your desktop using the [http://en.wikipedia.org/wiki/X_Window_System X Window system (X11)].
#* Requires an ''X11 Server'' application running on your client machine, and SSH X11 forwarding.
#* The X11 protocol can be slow over LAN.
# Run on Carbon within a [[HPC/Network Access/Virtual Desktop | virtual desktop environment (VNC)]].
#* Requires a VNC client on your machine.
#* It can be confusing where VNC and the app runs.
#* The virtual desktop presents a (deliberatly) sparse windowing environment – Copy&Paste can be difficult to use.
# Run on your Argonne-provided computer, checking out Carbon's license remotely.
#* Requires a full installation of COMSOL on your desktoo or laptop computer.
#* You will require access to the COMSOL license server running on Carbon, either through VPN (fairly easy) or SSH tunneling (can be intimidating for first-time users). See below for details.

=== Running a batch calculation ===
When the computational stage of models takes longer than a few minutes to complete, it will be beneficial to submit save the model file and run it in parallel on Carbon.
See the [[#Batch jobs|section below]].

== Use with X11 ==
For casual use, and with [[HPC/Network Access/SSH Tunnel Setup on Linux and MacOS | X11 tunneling to Carbon]] set up, simply add to your <code>~/.bashrc</code> file on Carbon:
module load comsol

Then, from the command line on a Carbon login node, start the COMSOL GUI:
comsol &

License configuration is automatic on Carbon.

Comsol will use a number of cores to run calculations multithreaded by default. (On Carbon, COMSOL_NUM_THREADS is set.)

'''Quit the application''' as soon as you are done. Only one user can use Comsol at a time.

== Use with VNC ==
See general [[HPC/Network Access/Virtual Desktop | instructions on using VNC]], and the COMSOL example therein.

== Native desktop version ==

To get the fastest response from the COMSOL GUI, run a native COMSOL version on your desktop machine or laptop, called ''client machine'' below.

Enter license details either in the COMSOL Setup configuration dialog.

== Native desktop version — License configuration ==
A client's COMSOL license needs to be configured depending on how the client reaches CNM network resources. Follow the appropriate section:

* [[#License configuration when inside CNM or on CNM VPN|License configuration when '''inside CNM''' or on '''CNM VPN''']]
: Requires that the user's machine is logically within the network of CNM/NST, either by a wired/auth WiFi connection in the building or by VPN under a CNM/NST profile.
* [[#License configuration when outside CNM – requires SSH Tunneling|License configuration when '''outside CNM''' – requires SSH Tunneling]]
: ''Non-NST staff'' usually have a VPN profile that places the machine logically [[#License configuration when outside CNM – requires SSH Tunneling|outside of NST's network.]]

== License configuration when inside CNM or on CNM VPN ==
=== Ensure correct DNS search domains ===
* Inspect the client machine's network configuration.
* Ensure that '''two''' entries are listed as DNS Search Domains for the current network profile:
nst.anl.gov
cnm.anl.gov
: (The order of entries does not matter for the purposes considered here.)
* If the client machine is ''wired to CNM networks'', the domains are normally provided correctly by DHCP.
* If the client machine is ''connected over VPN'', you may need edit the respective network profile.
:* On a Mac, find the VPN configuration details under System Preferences, Network, Advanced… . You may need to unlock the Network panel first before you can make changes under "Advanced…". It appears that those changes are being saved and will be activated automatically the next time that this VPN session gets started.
: [[Image:VNC_add_cnm_to_DNS_1.png|400px]] [[Image:VNC_add_cnm_to_DNS_2.png|400px]]
:* On Windows or Linux machines, take ''analogous'' steps, using the platform's own configuration tools.

* Exclude license manager components from client desktop installation. (Those are all handled server-side.)

=== License format ===
* Floating Network License (FNL), "Three-server redundancy"

=== License servers ===
* On client machines, while running the COMSOL installer or license configuration, specify license servers and ports as follows:
1718 clicense1
1718 clicense2
1718 clicense3
: '''Note:''' Use ''short host names only''. (Technical Reason: The license must be usable both on Carbon's internal network as well as on divisional networks).
1718 clicense1<strike>.cnm.anl.gov</strike>
1718 clicense2<strike>.cnm.anl.gov</strike>
1718 clicense3<strike>.cnm.anl.gov</strike>

'''Advanced:''' A client machine's COMSOL license file <code><COMSOLDIR>/Multiphysics/license/license.dat</code> should look as follows:
<source lang="make">
SERVER clicense1 ANY 1718
SERVER clicense2 ANY 1718
SERVER clicense3 ANY 1718
USE_SERVER
</source>

<code><COMSOLDIR></code> is located by default at:
* '''Windows''': C:\Program files\COMSOL\COMSOL''nn''
* '''Linux''': /usr/local/comsol''nn''
* '''Mac''': /Applications/COMSOL''nn''
("''nn''" is the application's abbreviated version number.)

=== Network Troubleshooting ===
When using the license access model descibed here (as opposed to tunneled over SSH), ensure the following:
* Verify that short host names can be resolved to IP addresses. Either one of the following commands should produce sensible output:
host clicense1
: or:
ping -c 3 clicense1
* Verify that ''both license ports on the same server'' can be reached from your client machine:
telnet '''clicense1''' 1718
telnet '''clicense1''' 1719
: or:
telnet '''clicense2''' 1718
telnet '''clicense2''' 1719
: A ''successful'' connection looks like:
Trying 146.139.x.y...
Connected to clicense1.cnm.anl.gov.
Escape character is '^]'.
:: Do not enter any data. Instead, close each connection by hitting <code>Ctrl</code><code>]</code> (closing square bracket), followed by <code>Ctrl</code><code>D</code>.
: A ''failed'' connection gives one of these errors:
telnet: connect to address 146.139.x.y: Operation timed out
: or:
telnet: connect to address 146.139.x.y: Connection refused
:: In these cases, ask that your IP firewall configuration be reviewed and changed.

== License configuration when outside CNM – requires SSH Tunneling ==
This section documents how to run COMSOL from a network location outside CNM.

As a one-time preparation step, two items must be configured:
* The license host that COMSOL is to contact, and
* An SSH connection that will allow your machine to access the COMSOL license server inside CNM's network.

To then ''run'' the COMSOL application, the configured SSH connection must be up and kept running for the entire time that you run the COMSOL. When you exit COMSOL and wish to start it again, you can keep the SSH connection running, it will be re-used for the next COMSOL run.

=== Configure COMSOL ===
The following needs to be done just once, typically during COMSOL installation:

In the COMSOL Installer, or in COMSOL Preferences, enter:
* '''License format:''' Floating Network License (FNL), "Port number and Hostname"
:: "Three-server redundancy" cannot be used because is cannot (easily) be accessed over SSH. Configure with just one server, and in rare tweak SSH to tunnel to the other server.
* '''Server:''' <code>127.0.0.1</code>
:: Do not use "localhost", as COMSOL interprets this incorrectly to mean that your machine ''is'' a license server.

* '''Port:''' <code>1718</code>


=== Establish connection to CNM license server ===
This step needs to be done prior to running COMSOL in your desktop session:
# Connect to <code>mega</code> with the following ssh command:
ssh -L 1718:'''clicense1''':1718 -L 1719:'''clicense1''':1719 mega.cnm.anl.gov
# Leave the command running between invocations of COMSOL.

=== Run COMSOL ===
# Ensure your connection to "mega" is still running. If not, restart it.
# Start the COMSOL application.

=== Troubleshooting ===
* If you get an error that the number of licenses is "exhausted", that is confirmation that you correctly reached the license server, though other users are using licenses.
* If the license access attempt takes a long time or hangs, the service might temporarily run on a secondary server. In this case, stop the mega session and restart it, connecting to the alternate license server:
ssh -L 1718:clicense'''2''':1718 -L 1719:clicense'''2''':1719 mega.cnm.anl.gov

== License Troubleshooting ==
The following are typical error messages.
=== Could not obtain license for COMSOL Multiphysics GUI ===
* Error details:
License error: -4.
'''Licensed number of users already reached.'''
Feature: COMSOLGUI
License path: /Users/stern/Applications/COMSOL52a/license/license.dat:
FlexNet Licensing error:-4,132
* Reason: The connection to the license server succeeded, but the license token(s) you requested are in use.
* Solution: Wait a bit, or [http://carbonmgmt/ganglia/lic.php identify who's using COMSOL] and politely ask the user when they might be finished with their work.

=== License server machine is down or not responding ===
* Error details:
License error: -96.
See the system administrator about starting the license server system, or
make sure you're referring to the right host (see LM_LICENSE_FILE).
Feature: SERIAL
Hostname: clicense3
License path: /Users/stern/Applications/COMSOL52a/license/license.dat:
FlexNet Licensing error:-96,7
* Reason: The license servers cannot be reached at all. The <code>ping</code> or <code>host</code> tests mentioned above fail.
* Solution: Review the client license file and DNS lookup domains.

=== License file does not support this version ===
* Error details:
Feature: SERIAL
'''Application version > License version: 5.21 > 3.5'''
License path: /Users/stern/Applications/COMSOL52a/license/license.dat:
FlexNet Licensing error:-21,126
* Reason: Only the first of two steps for connecting to the license servers succeeds, which may happen when the redundant license service runs on the alternate server instead of the one you're connected to.
* Solution: When using SSH, try <code>clicense2</code> instead of <code>clicense1</code>, or vice versa. Revisit your <code>~/.ssh/config</code> file or use the alternate command line as shown in the SSH section above.

== Multiple CPU cores ==
Ways to start multi-core (from http://www.comsol.com/support/knowledgebase/1096/)
comsol -np 8

export COMSOL_NUM_THREADS=8 # automatically provided on ''Carbon''
comsol
* COMSOL does not benefit from hyperthreading (oversubscribing the physical core count).

== Batch jobs ==

* http://www.comsol.com/support/knowledgebase/1001/ .
* [[Media:HPC 2012-05-16 Comsol parallel usage.pdf]] (Modified from http://www.comsol.com/support/knowledgebase/1001/files/4027/cluster_install_linux_42a.pptx – Appendix slide, PBS)
* '''Note:''' MPD is not supported on Carbon (I patched the main comsol script), and no longer necessary at all in Comsol-4.3.

=== Sample Job file ===
See:
$COMSOL_HOME/sample.job

== Important COMSOL Support Articles ==
* Toplevel suport page: https://www.comsol.com/support

=== KB Articles ===
* [https://www.comsol.com/support/knowledgebase/1001 Running COMSOL® in parallel on clusters]
* [https://www.comsol.com/support/knowledgebase/1250 Running parametric sweeps, batch sweeps, and cluster sweeps from the command line]
* [https://www.comsol.com/support/knowledgebase/1255 Reducing the amount of solution data stored in a model]
* [https://www.comsol.com/support/knowledgebase/830 Handling large models]

=== Blog-style articles ===
* [https://www.comsol.com/blogs/the-power-of-the-batch-sweep/ The Power of the Batch Sweep]
* [https://www.comsol.com/blogs/how-to-run-simulations-in-batch-mode-from-the-command-line/ How to Run Simulations in Batch Mode from the Command Line]

HPC/Applications/comsol

2026-02-26T16:58:15Z

Stern: /* License configuration when outside CNM – requires SSH Tunneling */

== Introduction ==
The [http://www.comsol.com/products/multiphysics/ '''COMSOL Multiphysics'''] application suite
is available for Mac, Windows, and Linux.

== Access ==
[[Image:HPC Comsol licensed modules.png|right|450px|thumb|COMSOL features licensed at CNM as of 2024-10.]]
=== COMSOL Multiphysics ===
Access to '''COMSOL Multiphysics''', the main application with both a graphical user interface and batch computing capabilities, will be granted '''''upon request''''' .

You must be:
* An '''''Argonne employee'''''  in one of the following roles:
** '''''NST staff member'''''  (including PostDocs, joint appointments, and students at NST).
** '''''CNM Facility User'''''  (including ''non-NST Argonne employees''), with an active proposal that has COMSOL in scope. Please see [[HPC/FAQ#Access_and_proposals]].

The set of available COMSOL modules that we have licensed is shown at right.

* The native desktop version can be installed only on ''Argonne-owned'' computers (laptop or desktop). Please contact [mailto:help@anl.gov?Subject=COMSOL%20license%20access%20request CNM support]. Make sure to review [http://www.comsol.com/support/updates/ Product Updates].
* To run COMSOL on our HPC system ''Carbon'', as GUI or as batch computing jobs, your proposal must have ''Carbon'' access in scope.

=== COMSOL Server ===
''CNM Facility Users'' who are not Argonne employees are not eligible to access
the full '''COMSOL Multiphysics''' product, in compliance with its licensing terms and conditions.
However, the [https://www.comsol.com/comsol-server '''COMSOL Server'''] product offers an alternative
albeit more narrow means of access.

All users, including '''''non-Argonne users''''' may be able to run, over a web interface in their browser,
pre-parameterized "fixed-model" simulations that were made with the '''COMSOL Application Builder''' (requires Windows).

To inquire about using COMSOL server applications,
please contact us.

We will review possible purchase paths for COMSOL Server, and the ''considerable effort'' likely required by users and the CNM to develop and deploy a COMSOL application.

== Support ==
* For questions on how to run or start COMSOL:
*: Submit an [[HPC/Support | '''IT support request''']]
* For questions related to most COMSOL data or scripts, contact COMSOL support at:
*: [mailto:support@comsol.com '''support'''@comsol.com]
* '''Exception:''' If your support question might contain data that is subject to '''US Export Control''' regulations, contact instead:
*: [mailto:us-export-control@us.comsol.com '''us-export-control'''@'''us'''.comsol.com]

== The modes to run COMSOL ==
There are several ways of running COMSOL.

=== Running the desktop application ===
To edit models and analyze results, use the COMSOL graphical user interface (GUI). You can run it in several ways, each explained in more detail in the following sections.

# Run on Carbon, display on your desktop using the [http://en.wikipedia.org/wiki/X_Window_System X Window system (X11)].
#* Requires an ''X11 Server'' application running on your client machine, and SSH X11 forwarding.
#* The X11 protocol can be slow over LAN.
# Run on Carbon within a [[HPC/Network Access/Virtual Desktop | virtual desktop environment (VNC)]].
#* Requires a VNC client on your machine.
#* It can be confusing where VNC and the app runs.
#* The virtual desktop presents a (deliberatly) sparse windowing environment – Copy&Paste can be difficult to use.
# Run on your Argonne-provided computer, checking out Carbon's license remotely.
#* Requires a full installation of COMSOL on your desktoo or laptop computer.
#* You will require access to the COMSOL license server running on Carbon, either through VPN (fairly easy) or SSH tunneling (can be intimidating for first-time users). See below for details.

=== Running a batch calculation ===
When the computational stage of models takes longer than a few minutes to complete, it will be beneficial to submit save the model file and run it in parallel on Carbon.
See the [[#Batch jobs|section below]].

== Use with X11 ==
For casual use, and with [[HPC/Network Access/SSH Tunnel Setup on Linux and MacOS | X11 tunneling to Carbon]] set up, simply add to your <code>~/.bashrc</code> file on Carbon:
module load comsol

Then, from the command line on a Carbon login node, start the COMSOL GUI:
comsol &

License configuration is automatic on Carbon.

Comsol will use a number of cores to run calculations multithreaded by default. (On Carbon, COMSOL_NUM_THREADS is set.)

'''Quit the application''' as soon as you are done. Only one user can use Comsol at a time.

== Use with VNC ==
See general [[HPC/Network Access/Virtual Desktop | instructions on using VNC]], and the COMSOL example therein.

== Native desktop version ==

To get the fastest response from the COMSOL GUI, run a native COMSOL version on your desktop machine or laptop, called ''client machine'' below.

Enter license details either in the COMSOL Setup configuration dialog.

== Native desktop version — License configuration ==
A client's COMSOL license needs to be configured depending on how the client reaches CNM network resources. Follow the appropriate section:

* [[#License configuration when inside CNM or on CNM VPN|License configuration when '''inside CNM''' or on '''CNM VPN''']]
: Requires that the user's machine is logically within the network of CNM/NST, either by a wired/auth WiFi connection in the building or by VPN under a CNM/NST profile.
* [[#License configuration when outside CNM – requires SSH Tunneling|License configuration when '''outside CNM''' – requires SSH Tunneling]]
: ''Non-NST staff'' usually have a VPN profile that places the machine logically [[#License configuration when outside CNM – requires SSH Tunneling|outside of NST's network.]]

== License configuration when inside CNM or on CNM VPN ==
=== Ensure correct DNS search domains ===
* Inspect the client machine's network configuration.
* Ensure that '''two''' entries are listed as DNS Search Domains for the current network profile:
nst.anl.gov
cnm.anl.gov
: (The order of entries does not matter for the purposes considered here.)
* If the client machine is ''wired to CNM networks'', the domains are normally provided correctly by DHCP.
* If the client machine is ''connected over VPN'', you may need edit the respective network profile.
:* On a Mac, find the VPN configuration details under System Preferences, Network, Advanced… . You may need to unlock the Network panel first before you can make changes under "Advanced…". It appears that those changes are being saved and will be activated automatically the next time that this VPN session gets started.
: [[Image:VNC_add_cnm_to_DNS_1.png|400px]] [[Image:VNC_add_cnm_to_DNS_2.png|400px]]
:* On Windows or Linux machines, take ''analogous'' steps, using the platform's own configuration tools.

* Exclude license manager components from client desktop installation. (Those are all handled server-side.)

=== License format ===
* Floating Network License (FNL), "Three-server redundancy"

=== License servers ===
* On client machines, while running the COMSOL installer or license configuration, specify license servers and ports as follows:
1718 clicense1
1718 clicense2
1718 clicense3
: '''Note:''' Use ''short host names only''. (Technical Reason: The license must be usable both on Carbon's internal network as well as on divisional networks).
1718 clicense1<strike>.cnm.anl.gov</strike>
1718 clicense2<strike>.cnm.anl.gov</strike>
1718 clicense3<strike>.cnm.anl.gov</strike>

'''Advanced:''' A client machine's COMSOL license file <code><COMSOLDIR>/Multiphysics/license/license.dat</code> should look as follows:
<source lang="make">
SERVER clicense1 ANY 1718
SERVER clicense2 ANY 1718
SERVER clicense3 ANY 1718
USE_SERVER
</source>

<code><COMSOLDIR></code> is located by default at:
* '''Windows''': C:\Program files\COMSOL\COMSOL''nn''
* '''Linux''': /usr/local/comsol''nn''
* '''Mac''': /Applications/COMSOL''nn''
("''nn''" is the application's abbreviated version number.)

=== Network Troubleshooting ===
When using the license access model descibed here (as opposed to tunneled over SSH), ensure the following:
* Verify that short host names can be resolved to IP addresses. Either one of the following commands should produce sensible output:
host clicense1
: or:
ping -c 3 clicense1
* Verify that ''both license ports on the same server'' can be reached from your client machine:
telnet '''clicense1''' 1718
telnet '''clicense1''' 1719
: or:
telnet '''clicense2''' 1718
telnet '''clicense2''' 1719
: A ''successful'' connection looks like:
Trying 146.139.x.y...
Connected to clicense1.cnm.anl.gov.
Escape character is '^]'.
:: Do not enter any data. Instead, close each connection by hitting <code>Ctrl</code><code>]</code> (closing square bracket), followed by <code>Ctrl</code><code>D</code>.
: A ''failed'' connection gives one of these errors:
telnet: connect to address 146.139.x.y: Operation timed out
: or:
telnet: connect to address 146.139.x.y: Connection refused
:: In these cases, ask that your IP firewall configuration be reviewed and changed.

== License configuration when outside CNM – requires SSH Tunneling ==
If you're comfortable with ssh and tunneling, set up client licensing as follows:

=== Configure COMSOL ===
The following needs to be done just once, typically during COMSOL installation:

In the COMSOL Installer, or in COMSOL Preferences, enter:
* '''License format:''' Floating Network License (FNL), "Port number and Hostname"
:: "Three-server redundancy" cannot be used because is cannot (easily) be accessed over SSH. Configure with just one server, and in rare tweak SSH to tunnel to the other server.
* '''Server:''' <code>127.0.0.1</code>
:: Do not use "localhost", as COMSOL interprets this incorrectly to mean that your machine ''is'' a license server.

* '''Port:''' <code>1718</code>


=== Establish connection to CNM license server ===
This step needs to be done prior to running COMSOL in your desktop session:
# Connect to <code>mega</code> with the following ssh command:
ssh -L 1718:'''clicense1''':1718 -L 1719:'''clicense1''':1719 mega.cnm.anl.gov
# Leave the command running between invocations of COMSOL.

=== Run COMSOL ===
# Ensure your connection to "mega" is still running. If not, restart it.
# Start the COMSOL application.

=== Troubleshooting ===
* If you get an error that the number of licenses is "exhausted", that is confirmation that you correctly reached the license server, though other users are using licenses.
* If the license access attempt takes a long time or hangs, the service might temporarily run on a secondary server. In this case, stop the mega session and restart it, connecting to the alternate license server:
ssh -L 1718:clicense'''2''':1718 -L 1719:clicense'''2''':1719 mega.cnm.anl.gov

== License Troubleshooting ==
The following are typical error messages.
=== Could not obtain license for COMSOL Multiphysics GUI ===
* Error details:
License error: -4.
'''Licensed number of users already reached.'''
Feature: COMSOLGUI
License path: /Users/stern/Applications/COMSOL52a/license/license.dat:
FlexNet Licensing error:-4,132
* Reason: The connection to the license server succeeded, but the license token(s) you requested are in use.
* Solution: Wait a bit, or [http://carbonmgmt/ganglia/lic.php identify who's using COMSOL] and politely ask the user when they might be finished with their work.

=== License server machine is down or not responding ===
* Error details:
License error: -96.
See the system administrator about starting the license server system, or
make sure you're referring to the right host (see LM_LICENSE_FILE).
Feature: SERIAL
Hostname: clicense3
License path: /Users/stern/Applications/COMSOL52a/license/license.dat:
FlexNet Licensing error:-96,7
* Reason: The license servers cannot be reached at all. The <code>ping</code> or <code>host</code> tests mentioned above fail.
* Solution: Review the client license file and DNS lookup domains.

=== License file does not support this version ===
* Error details:
Feature: SERIAL
'''Application version > License version: 5.21 > 3.5'''
License path: /Users/stern/Applications/COMSOL52a/license/license.dat:
FlexNet Licensing error:-21,126
* Reason: Only the first of two steps for connecting to the license servers succeeds, which may happen when the redundant license service runs on the alternate server instead of the one you're connected to.
* Solution: When using SSH, try <code>clicense2</code> instead of <code>clicense1</code>, or vice versa. Revisit your <code>~/.ssh/config</code> file or use the alternate command line as shown in the SSH section above.

== Multiple CPU cores ==
Ways to start multi-core (from http://www.comsol.com/support/knowledgebase/1096/)
comsol -np 8

export COMSOL_NUM_THREADS=8 # automatically provided on ''Carbon''
comsol
* COMSOL does not benefit from hyperthreading (oversubscribing the physical core count).

== Batch jobs ==

* http://www.comsol.com/support/knowledgebase/1001/ .
* [[Media:HPC 2012-05-16 Comsol parallel usage.pdf]] (Modified from http://www.comsol.com/support/knowledgebase/1001/files/4027/cluster_install_linux_42a.pptx – Appendix slide, PBS)
* '''Note:''' MPD is not supported on Carbon (I patched the main comsol script), and no longer necessary at all in Comsol-4.3.

=== Sample Job file ===
See:
$COMSOL_HOME/sample.job

== Important COMSOL Support Articles ==
* Toplevel suport page: https://www.comsol.com/support

=== KB Articles ===
* [https://www.comsol.com/support/knowledgebase/1001 Running COMSOL® in parallel on clusters]
* [https://www.comsol.com/support/knowledgebase/1250 Running parametric sweeps, batch sweeps, and cluster sweeps from the command line]
* [https://www.comsol.com/support/knowledgebase/1255 Reducing the amount of solution data stored in a model]
* [https://www.comsol.com/support/knowledgebase/830 Handling large models]

=== Blog-style articles ===
* [https://www.comsol.com/blogs/the-power-of-the-batch-sweep/ The Power of the Batch Sweep]
* [https://www.comsol.com/blogs/how-to-run-simulations-in-batch-mode-from-the-command-line/ How to Run Simulations in Batch Mode from the Command Line]

HPC/Applications/comsol

2026-02-26T16:57:34Z

Stern: /* License configuration when outside CNM – requires SSH Tunneling */

== Introduction ==
The [http://www.comsol.com/products/multiphysics/ '''COMSOL Multiphysics'''] application suite
is available for Mac, Windows, and Linux.

== Access ==
[[Image:HPC Comsol licensed modules.png|right|450px|thumb|COMSOL features licensed at CNM as of 2024-10.]]
=== COMSOL Multiphysics ===
Access to '''COMSOL Multiphysics''', the main application with both a graphical user interface and batch computing capabilities, will be granted '''''upon request''''' .

You must be:
* An '''''Argonne employee'''''  in one of the following roles:
** '''''NST staff member'''''  (including PostDocs, joint appointments, and students at NST).
** '''''CNM Facility User'''''  (including ''non-NST Argonne employees''), with an active proposal that has COMSOL in scope. Please see [[HPC/FAQ#Access_and_proposals]].

The set of available COMSOL modules that we have licensed is shown at right.

* The native desktop version can be installed only on ''Argonne-owned'' computers (laptop or desktop). Please contact [mailto:help@anl.gov?Subject=COMSOL%20license%20access%20request CNM support]. Make sure to review [http://www.comsol.com/support/updates/ Product Updates].
* To run COMSOL on our HPC system ''Carbon'', as GUI or as batch computing jobs, your proposal must have ''Carbon'' access in scope.

=== COMSOL Server ===
''CNM Facility Users'' who are not Argonne employees are not eligible to access
the full '''COMSOL Multiphysics''' product, in compliance with its licensing terms and conditions.
However, the [https://www.comsol.com/comsol-server '''COMSOL Server'''] product offers an alternative
albeit more narrow means of access.

All users, including '''''non-Argonne users''''' may be able to run, over a web interface in their browser,
pre-parameterized "fixed-model" simulations that were made with the '''COMSOL Application Builder''' (requires Windows).

To inquire about using COMSOL server applications,
please contact us.

We will review possible purchase paths for COMSOL Server, and the ''considerable effort'' likely required by users and the CNM to develop and deploy a COMSOL application.

== Support ==
* For questions on how to run or start COMSOL:
*: Submit an [[HPC/Support | '''IT support request''']]
* For questions related to most COMSOL data or scripts, contact COMSOL support at:
*: [mailto:support@comsol.com '''support'''@comsol.com]
* '''Exception:''' If your support question might contain data that is subject to '''US Export Control''' regulations, contact instead:
*: [mailto:us-export-control@us.comsol.com '''us-export-control'''@'''us'''.comsol.com]

== The modes to run COMSOL ==
There are several ways of running COMSOL.

=== Running the desktop application ===
To edit models and analyze results, use the COMSOL graphical user interface (GUI). You can run it in several ways, each explained in more detail in the following sections.

# Run on Carbon, display on your desktop using the [http://en.wikipedia.org/wiki/X_Window_System X Window system (X11)].
#* Requires an ''X11 Server'' application running on your client machine, and SSH X11 forwarding.
#* The X11 protocol can be slow over LAN.
# Run on Carbon within a [[HPC/Network Access/Virtual Desktop | virtual desktop environment (VNC)]].
#* Requires a VNC client on your machine.
#* It can be confusing where VNC and the app runs.
#* The virtual desktop presents a (deliberatly) sparse windowing environment – Copy&Paste can be difficult to use.
# Run on your Argonne-provided computer, checking out Carbon's license remotely.
#* Requires a full installation of COMSOL on your desktoo or laptop computer.
#* You will require access to the COMSOL license server running on Carbon, either through VPN (fairly easy) or SSH tunneling (can be intimidating for first-time users). See below for details.

=== Running a batch calculation ===
When the computational stage of models takes longer than a few minutes to complete, it will be beneficial to submit save the model file and run it in parallel on Carbon.
See the [[#Batch jobs|section below]].

== Use with X11 ==
For casual use, and with [[HPC/Network Access/SSH Tunnel Setup on Linux and MacOS | X11 tunneling to Carbon]] set up, simply add to your <code>~/.bashrc</code> file on Carbon:
module load comsol

Then, from the command line on a Carbon login node, start the COMSOL GUI:
comsol &

License configuration is automatic on Carbon.

Comsol will use a number of cores to run calculations multithreaded by default. (On Carbon, COMSOL_NUM_THREADS is set.)

'''Quit the application''' as soon as you are done. Only one user can use Comsol at a time.

== Use with VNC ==
See general [[HPC/Network Access/Virtual Desktop | instructions on using VNC]], and the COMSOL example therein.

== Native desktop version ==

To get the fastest response from the COMSOL GUI, run a native COMSOL version on your desktop machine or laptop, called ''client machine'' below.

Enter license details either in the COMSOL Setup configuration dialog.

== Native desktop version — License configuration ==
A client's COMSOL license needs to be configured depending on how the client reaches CNM network resources. Follow the appropriate section:

* [[#License configuration when inside CNM or on CNM VPN|License configuration when '''inside CNM''' or on '''CNM VPN''']]
: Requires that the user's machine is logically within the network of CNM/NST, either by a wired/auth WiFi connection in the building or by VPN under a CNM/NST profile.
* [[#License configuration when outside CNM – requires SSH Tunneling|License configuration when '''outside CNM''' – requires SSH Tunneling]]
: ''Non-NST staff'' usually have a VPN profile that places the machine logically [[#License configuration when outside CNM – requires SSH Tunneling|outside of NST's network.]]

== License configuration when inside CNM or on CNM VPN ==
=== Ensure correct DNS search domains ===
* Inspect the client machine's network configuration.
* Ensure that '''two''' entries are listed as DNS Search Domains for the current network profile:
nst.anl.gov
cnm.anl.gov
: (The order of entries does not matter for the purposes considered here.)
* If the client machine is ''wired to CNM networks'', the domains are normally provided correctly by DHCP.
* If the client machine is ''connected over VPN'', you may need edit the respective network profile.
:* On a Mac, find the VPN configuration details under System Preferences, Network, Advanced… . You may need to unlock the Network panel first before you can make changes under "Advanced…". It appears that those changes are being saved and will be activated automatically the next time that this VPN session gets started.
: [[Image:VNC_add_cnm_to_DNS_1.png|400px]] [[Image:VNC_add_cnm_to_DNS_2.png|400px]]
:* On Windows or Linux machines, take ''analogous'' steps, using the platform's own configuration tools.

* Exclude license manager components from client desktop installation. (Those are all handled server-side.)

=== License format ===
* Floating Network License (FNL), "Three-server redundancy"

=== License servers ===
* On client machines, while running the COMSOL installer or license configuration, specify license servers and ports as follows:
1718 clicense1
1718 clicense2
1718 clicense3
: '''Note:''' Use ''short host names only''. (Technical Reason: The license must be usable both on Carbon's internal network as well as on divisional networks).
1718 clicense1<strike>.cnm.anl.gov</strike>
1718 clicense2<strike>.cnm.anl.gov</strike>
1718 clicense3<strike>.cnm.anl.gov</strike>

'''Advanced:''' A client machine's COMSOL license file <code><COMSOLDIR>/Multiphysics/license/license.dat</code> should look as follows:
<source lang="make">
SERVER clicense1 ANY 1718
SERVER clicense2 ANY 1718
SERVER clicense3 ANY 1718
USE_SERVER
</source>

<code><COMSOLDIR></code> is located by default at:
* '''Windows''': C:\Program files\COMSOL\COMSOL''nn''
* '''Linux''': /usr/local/comsol''nn''
* '''Mac''': /Applications/COMSOL''nn''
("''nn''" is the application's abbreviated version number.)

=== Network Troubleshooting ===
When using the license access model descibed here (as opposed to tunneled over SSH), ensure the following:
* Verify that short host names can be resolved to IP addresses. Either one of the following commands should produce sensible output:
host clicense1
: or:
ping -c 3 clicense1
* Verify that ''both license ports on the same server'' can be reached from your client machine:
telnet '''clicense1''' 1718
telnet '''clicense1''' 1719
: or:
telnet '''clicense2''' 1718
telnet '''clicense2''' 1719
: A ''successful'' connection looks like:
Trying 146.139.x.y...
Connected to clicense1.cnm.anl.gov.
Escape character is '^]'.
:: Do not enter any data. Instead, close each connection by hitting <code>Ctrl</code><code>]</code> (closing square bracket), followed by <code>Ctrl</code><code>D</code>.
: A ''failed'' connection gives one of these errors:
telnet: connect to address 146.139.x.y: Operation timed out
: or:
telnet: connect to address 146.139.x.y: Connection refused
:: In these cases, ask that your IP firewall configuration be reviewed and changed.

== License configuration when outside CNM – requires SSH Tunneling ==
If you're comfortable with ssh and tunneling, set up client licensing as follows:

=== Configure COMSOL ===
The following needs to be done just once, typically during COMSOL installation:

In the COMSOL Installer, or in COMSOL Preferences, enter:
* '''License format:''' Floating Network License (FNL), "Port number and Hostname"
:: "Three-server redundancy" cannot be used because is cannot (easily) be accessed over SSH. Configure with just one server, and in rare tweak SSH to tunnel to the other server.
* '''Server:''' <code>127.0.0.1</code>
:: Do not use "localhost", as COMSOL interprets this incorrectly to mean that your machine ''is'' a license server.

* '''Port:''' <code>1718</code>


=== Establish connection to CNM license server ===
This step needs to be done prior to running COMSOL in your desktop session:
# Connect to <code>mega</code> with the following ssh command:
ssh -L 1718:'''clicense1''':1718 -L 1719:'''clicense1''':1719 mega.cnm.anl.gov
# Leave the command running between invocations of COMSOL.

=== Run COMSOL ===
# Ensure your connection to "mega" is still running. If not, restart it.
# Start the COMSOL application.
#* If you get an error that the number of licenses is "exhausted", that is confirmation that you correctly reached the license server, though other users are using licenses.
#* If the license access attempt takes a long time or hangs, the service might temporarily run on a secondary server. In this case, stop the mega session and restart it, connecting to the alternate license server:
ssh -L 1718:clicense'''2''':1718 -L 1719:clicense'''2''':1719 mega.cnm.anl.gov

== License Troubleshooting ==
The following are typical error messages.
=== Could not obtain license for COMSOL Multiphysics GUI ===
* Error details:
License error: -4.
'''Licensed number of users already reached.'''
Feature: COMSOLGUI
License path: /Users/stern/Applications/COMSOL52a/license/license.dat:
FlexNet Licensing error:-4,132
* Reason: The connection to the license server succeeded, but the license token(s) you requested are in use.
* Solution: Wait a bit, or [http://carbonmgmt/ganglia/lic.php identify who's using COMSOL] and politely ask the user when they might be finished with their work.

=== License server machine is down or not responding ===
* Error details:
License error: -96.
See the system administrator about starting the license server system, or
make sure you're referring to the right host (see LM_LICENSE_FILE).
Feature: SERIAL
Hostname: clicense3
License path: /Users/stern/Applications/COMSOL52a/license/license.dat:
FlexNet Licensing error:-96,7
* Reason: The license servers cannot be reached at all. The <code>ping</code> or <code>host</code> tests mentioned above fail.
* Solution: Review the client license file and DNS lookup domains.

=== License file does not support this version ===
* Error details:
Feature: SERIAL
'''Application version > License version: 5.21 > 3.5'''
License path: /Users/stern/Applications/COMSOL52a/license/license.dat:
FlexNet Licensing error:-21,126
* Reason: Only the first of two steps for connecting to the license servers succeeds, which may happen when the redundant license service runs on the alternate server instead of the one you're connected to.
* Solution: When using SSH, try <code>clicense2</code> instead of <code>clicense1</code>, or vice versa. Revisit your <code>~/.ssh/config</code> file or use the alternate command line as shown in the SSH section above.

== Multiple CPU cores ==
Ways to start multi-core (from http://www.comsol.com/support/knowledgebase/1096/)
comsol -np 8

export COMSOL_NUM_THREADS=8 # automatically provided on ''Carbon''
comsol
* COMSOL does not benefit from hyperthreading (oversubscribing the physical core count).

== Batch jobs ==

* http://www.comsol.com/support/knowledgebase/1001/ .
* [[Media:HPC 2012-05-16 Comsol parallel usage.pdf]] (Modified from http://www.comsol.com/support/knowledgebase/1001/files/4027/cluster_install_linux_42a.pptx – Appendix slide, PBS)
* '''Note:''' MPD is not supported on Carbon (I patched the main comsol script), and no longer necessary at all in Comsol-4.3.

=== Sample Job file ===
See:
$COMSOL_HOME/sample.job

== Important COMSOL Support Articles ==
* Toplevel suport page: https://www.comsol.com/support

=== KB Articles ===
* [https://www.comsol.com/support/knowledgebase/1001 Running COMSOL® in parallel on clusters]
* [https://www.comsol.com/support/knowledgebase/1250 Running parametric sweeps, batch sweeps, and cluster sweeps from the command line]
* [https://www.comsol.com/support/knowledgebase/1255 Reducing the amount of solution data stored in a model]
* [https://www.comsol.com/support/knowledgebase/830 Handling large models]

=== Blog-style articles ===
* [https://www.comsol.com/blogs/the-power-of-the-batch-sweep/ The Power of the Batch Sweep]
* [https://www.comsol.com/blogs/how-to-run-simulations-in-batch-mode-from-the-command-line/ How to Run Simulations in Batch Mode from the Command Line]

HPC/Applications/comsol

2026-02-26T16:56:17Z

Stern: /* License configuration when outside CNM – requires SSH Tunneling */

== Introduction ==
The [http://www.comsol.com/products/multiphysics/ '''COMSOL Multiphysics'''] application suite
is available for Mac, Windows, and Linux.

== Access ==
[[Image:HPC Comsol licensed modules.png|right|450px|thumb|COMSOL features licensed at CNM as of 2024-10.]]
=== COMSOL Multiphysics ===
Access to '''COMSOL Multiphysics''', the main application with both a graphical user interface and batch computing capabilities, will be granted '''''upon request''''' .

You must be:
* An '''''Argonne employee'''''  in one of the following roles:
** '''''NST staff member'''''  (including PostDocs, joint appointments, and students at NST).
** '''''CNM Facility User'''''  (including ''non-NST Argonne employees''), with an active proposal that has COMSOL in scope. Please see [[HPC/FAQ#Access_and_proposals]].

The set of available COMSOL modules that we have licensed is shown at right.

* The native desktop version can be installed only on ''Argonne-owned'' computers (laptop or desktop). Please contact [mailto:help@anl.gov?Subject=COMSOL%20license%20access%20request CNM support]. Make sure to review [http://www.comsol.com/support/updates/ Product Updates].
* To run COMSOL on our HPC system ''Carbon'', as GUI or as batch computing jobs, your proposal must have ''Carbon'' access in scope.

=== COMSOL Server ===
''CNM Facility Users'' who are not Argonne employees are not eligible to access
the full '''COMSOL Multiphysics''' product, in compliance with its licensing terms and conditions.
However, the [https://www.comsol.com/comsol-server '''COMSOL Server'''] product offers an alternative
albeit more narrow means of access.

All users, including '''''non-Argonne users''''' may be able to run, over a web interface in their browser,
pre-parameterized "fixed-model" simulations that were made with the '''COMSOL Application Builder''' (requires Windows).

To inquire about using COMSOL server applications,
please contact us.

We will review possible purchase paths for COMSOL Server, and the ''considerable effort'' likely required by users and the CNM to develop and deploy a COMSOL application.

== Support ==
* For questions on how to run or start COMSOL:
*: Submit an [[HPC/Support | '''IT support request''']]
* For questions related to most COMSOL data or scripts, contact COMSOL support at:
*: [mailto:support@comsol.com '''support'''@comsol.com]
* '''Exception:''' If your support question might contain data that is subject to '''US Export Control''' regulations, contact instead:
*: [mailto:us-export-control@us.comsol.com '''us-export-control'''@'''us'''.comsol.com]

== The modes to run COMSOL ==
There are several ways of running COMSOL.

=== Running the desktop application ===
To edit models and analyze results, use the COMSOL graphical user interface (GUI). You can run it in several ways, each explained in more detail in the following sections.

# Run on Carbon, display on your desktop using the [http://en.wikipedia.org/wiki/X_Window_System X Window system (X11)].
#* Requires an ''X11 Server'' application running on your client machine, and SSH X11 forwarding.
#* The X11 protocol can be slow over LAN.
# Run on Carbon within a [[HPC/Network Access/Virtual Desktop | virtual desktop environment (VNC)]].
#* Requires a VNC client on your machine.
#* It can be confusing where VNC and the app runs.
#* The virtual desktop presents a (deliberatly) sparse windowing environment – Copy&Paste can be difficult to use.
# Run on your Argonne-provided computer, checking out Carbon's license remotely.
#* Requires a full installation of COMSOL on your desktoo or laptop computer.
#* You will require access to the COMSOL license server running on Carbon, either through VPN (fairly easy) or SSH tunneling (can be intimidating for first-time users). See below for details.

=== Running a batch calculation ===
When the computational stage of models takes longer than a few minutes to complete, it will be beneficial to submit save the model file and run it in parallel on Carbon.
See the [[#Batch jobs|section below]].

== Use with X11 ==
For casual use, and with [[HPC/Network Access/SSH Tunnel Setup on Linux and MacOS | X11 tunneling to Carbon]] set up, simply add to your <code>~/.bashrc</code> file on Carbon:
module load comsol

Then, from the command line on a Carbon login node, start the COMSOL GUI:
comsol &

License configuration is automatic on Carbon.

Comsol will use a number of cores to run calculations multithreaded by default. (On Carbon, COMSOL_NUM_THREADS is set.)

'''Quit the application''' as soon as you are done. Only one user can use Comsol at a time.

== Use with VNC ==
See general [[HPC/Network Access/Virtual Desktop | instructions on using VNC]], and the COMSOL example therein.

== Native desktop version ==

To get the fastest response from the COMSOL GUI, run a native COMSOL version on your desktop machine or laptop, called ''client machine'' below.

Enter license details either in the COMSOL Setup configuration dialog.

== Native desktop version — License configuration ==
A client's COMSOL license needs to be configured depending on how the client reaches CNM network resources. Follow the appropriate section:

* [[#License configuration when inside CNM or on CNM VPN|License configuration when '''inside CNM''' or on '''CNM VPN''']]
: Requires that the user's machine is logically within the network of CNM/NST, either by a wired/auth WiFi connection in the building or by VPN under a CNM/NST profile.
* [[#License configuration when outside CNM – requires SSH Tunneling|License configuration when '''outside CNM''' – requires SSH Tunneling]]
: ''Non-NST staff'' usually have a VPN profile that places the machine logically [[#License configuration when outside CNM – requires SSH Tunneling|outside of NST's network.]]

== License configuration when inside CNM or on CNM VPN ==
=== Ensure correct DNS search domains ===
* Inspect the client machine's network configuration.
* Ensure that '''two''' entries are listed as DNS Search Domains for the current network profile:
nst.anl.gov
cnm.anl.gov
: (The order of entries does not matter for the purposes considered here.)
* If the client machine is ''wired to CNM networks'', the domains are normally provided correctly by DHCP.
* If the client machine is ''connected over VPN'', you may need edit the respective network profile.
:* On a Mac, find the VPN configuration details under System Preferences, Network, Advanced… . You may need to unlock the Network panel first before you can make changes under "Advanced…". It appears that those changes are being saved and will be activated automatically the next time that this VPN session gets started.
: [[Image:VNC_add_cnm_to_DNS_1.png|400px]] [[Image:VNC_add_cnm_to_DNS_2.png|400px]]
:* On Windows or Linux machines, take ''analogous'' steps, using the platform's own configuration tools.

* Exclude license manager components from client desktop installation. (Those are all handled server-side.)

=== License format ===
* Floating Network License (FNL), "Three-server redundancy"

=== License servers ===
* On client machines, while running the COMSOL installer or license configuration, specify license servers and ports as follows:
1718 clicense1
1718 clicense2
1718 clicense3
: '''Note:''' Use ''short host names only''. (Technical Reason: The license must be usable both on Carbon's internal network as well as on divisional networks).
1718 clicense1<strike>.cnm.anl.gov</strike>
1718 clicense2<strike>.cnm.anl.gov</strike>
1718 clicense3<strike>.cnm.anl.gov</strike>

'''Advanced:''' A client machine's COMSOL license file <code><COMSOLDIR>/Multiphysics/license/license.dat</code> should look as follows:
<source lang="make">
SERVER clicense1 ANY 1718
SERVER clicense2 ANY 1718
SERVER clicense3 ANY 1718
USE_SERVER
</source>

<code><COMSOLDIR></code> is located by default at:
* '''Windows''': C:\Program files\COMSOL\COMSOL''nn''
* '''Linux''': /usr/local/comsol''nn''
* '''Mac''': /Applications/COMSOL''nn''
("''nn''" is the application's abbreviated version number.)

=== Network Troubleshooting ===
When using the license access model descibed here (as opposed to tunneled over SSH), ensure the following:
* Verify that short host names can be resolved to IP addresses. Either one of the following commands should produce sensible output:
host clicense1
: or:
ping -c 3 clicense1
* Verify that ''both license ports on the same server'' can be reached from your client machine:
telnet '''clicense1''' 1718
telnet '''clicense1''' 1719
: or:
telnet '''clicense2''' 1718
telnet '''clicense2''' 1719
: A ''successful'' connection looks like:
Trying 146.139.x.y...
Connected to clicense1.cnm.anl.gov.
Escape character is '^]'.
:: Do not enter any data. Instead, close each connection by hitting <code>Ctrl</code><code>]</code> (closing square bracket), followed by <code>Ctrl</code><code>D</code>.
: A ''failed'' connection gives one of these errors:
telnet: connect to address 146.139.x.y: Operation timed out
: or:
telnet: connect to address 146.139.x.y: Connection refused
:: In these cases, ask that your IP firewall configuration be reviewed and changed.

== License configuration when outside CNM – requires SSH Tunneling ==
If you're comfortable with ssh and tunneling, set up client licensing as follows:

=== Configure COMSOL ===
The following needs to be done just once, typically during COMSOL installation:

In the COMSOL Installer, or in COMSOL Preferences, enter:
* '''License format:''' Floating Network License (FNL), "Port number and Hostname"
:: "Three-server redundancy" cannot be used because is cannot (easily) be accessed over SSH. Configure with just one server, and in rare tweak SSH to tunnel to the other server.
* '''Server:''' <code>127.0.0.1</code>
:: Do not use "localhost", as COMSOL interprets this incorrectly to mean that your machine ''is'' a license server.

* '''Port:''' <code>1718</code>


=== Establish connection to CNM license server ===
This step needs to be done prior to running COMSOL in your desktop session:
* Connect to <code>mega</code> with the following ssh command:
ssh -L 1718:'''clicense1''':1718 -L 1719:'''clicense1''':1719 mega.cnm.anl.gov
* Leave the command running between invocations of COMSOL.

=== Run COMSOL when outside CNM ===
* Ensure your connection to "mega" is still running. If not, restart it.
* Start the COMSOL application.

** If you get an error that the number of licenses is "exhausted", that is confirmation that you correctly reached the license server, though other users are using licenses.
** If the license access attempt takes a long time or hangs, the service might temporarily run on a secondary server. In this case, stop the mega session and restart it, connecting to the alternate license server:
ssh -L 1718:clicense'''2''':1718 -L 1719:clicense'''2''':1719 mega.cnm.anl.gov

== License Troubleshooting ==
The following are typical error messages.
=== Could not obtain license for COMSOL Multiphysics GUI ===
* Error details:
License error: -4.
'''Licensed number of users already reached.'''
Feature: COMSOLGUI
License path: /Users/stern/Applications/COMSOL52a/license/license.dat:
FlexNet Licensing error:-4,132
* Reason: The connection to the license server succeeded, but the license token(s) you requested are in use.
* Solution: Wait a bit, or [http://carbonmgmt/ganglia/lic.php identify who's using COMSOL] and politely ask the user when they might be finished with their work.

=== License server machine is down or not responding ===
* Error details:
License error: -96.
See the system administrator about starting the license server system, or
make sure you're referring to the right host (see LM_LICENSE_FILE).
Feature: SERIAL
Hostname: clicense3
License path: /Users/stern/Applications/COMSOL52a/license/license.dat:
FlexNet Licensing error:-96,7
* Reason: The license servers cannot be reached at all. The <code>ping</code> or <code>host</code> tests mentioned above fail.
* Solution: Review the client license file and DNS lookup domains.

=== License file does not support this version ===
* Error details:
Feature: SERIAL
'''Application version > License version: 5.21 > 3.5'''
License path: /Users/stern/Applications/COMSOL52a/license/license.dat:
FlexNet Licensing error:-21,126
* Reason: Only the first of two steps for connecting to the license servers succeeds, which may happen when the redundant license service runs on the alternate server instead of the one you're connected to.
* Solution: When using SSH, try <code>clicense2</code> instead of <code>clicense1</code>, or vice versa. Revisit your <code>~/.ssh/config</code> file or use the alternate command line as shown in the SSH section above.

== Multiple CPU cores ==
Ways to start multi-core (from http://www.comsol.com/support/knowledgebase/1096/)
comsol -np 8

export COMSOL_NUM_THREADS=8 # automatically provided on ''Carbon''
comsol
* COMSOL does not benefit from hyperthreading (oversubscribing the physical core count).

== Batch jobs ==

* http://www.comsol.com/support/knowledgebase/1001/ .
* [[Media:HPC 2012-05-16 Comsol parallel usage.pdf]] (Modified from http://www.comsol.com/support/knowledgebase/1001/files/4027/cluster_install_linux_42a.pptx – Appendix slide, PBS)
* '''Note:''' MPD is not supported on Carbon (I patched the main comsol script), and no longer necessary at all in Comsol-4.3.

=== Sample Job file ===
See:
$COMSOL_HOME/sample.job

== Important COMSOL Support Articles ==
* Toplevel suport page: https://www.comsol.com/support

=== KB Articles ===
* [https://www.comsol.com/support/knowledgebase/1001 Running COMSOL® in parallel on clusters]
* [https://www.comsol.com/support/knowledgebase/1250 Running parametric sweeps, batch sweeps, and cluster sweeps from the command line]
* [https://www.comsol.com/support/knowledgebase/1255 Reducing the amount of solution data stored in a model]
* [https://www.comsol.com/support/knowledgebase/830 Handling large models]

=== Blog-style articles ===
* [https://www.comsol.com/blogs/the-power-of-the-batch-sweep/ The Power of the Batch Sweep]
* [https://www.comsol.com/blogs/how-to-run-simulations-in-batch-mode-from-the-command-line/ How to Run Simulations in Batch Mode from the Command Line]

HPC/Applications/comsol

2026-02-26T16:54:55Z

Stern: /* License configuration when inside CNM or on CNM VPN */

== Introduction ==
The [http://www.comsol.com/products/multiphysics/ '''COMSOL Multiphysics'''] application suite
is available for Mac, Windows, and Linux.

== Access ==
[[Image:HPC Comsol licensed modules.png|right|450px|thumb|COMSOL features licensed at CNM as of 2024-10.]]
=== COMSOL Multiphysics ===
Access to '''COMSOL Multiphysics''', the main application with both a graphical user interface and batch computing capabilities, will be granted '''''upon request''''' .

You must be:
* An '''''Argonne employee'''''  in one of the following roles:
** '''''NST staff member'''''  (including PostDocs, joint appointments, and students at NST).
** '''''CNM Facility User'''''  (including ''non-NST Argonne employees''), with an active proposal that has COMSOL in scope. Please see [[HPC/FAQ#Access_and_proposals]].

The set of available COMSOL modules that we have licensed is shown at right.

* The native desktop version can be installed only on ''Argonne-owned'' computers (laptop or desktop). Please contact [mailto:help@anl.gov?Subject=COMSOL%20license%20access%20request CNM support]. Make sure to review [http://www.comsol.com/support/updates/ Product Updates].
* To run COMSOL on our HPC system ''Carbon'', as GUI or as batch computing jobs, your proposal must have ''Carbon'' access in scope.

=== COMSOL Server ===
''CNM Facility Users'' who are not Argonne employees are not eligible to access
the full '''COMSOL Multiphysics''' product, in compliance with its licensing terms and conditions.
However, the [https://www.comsol.com/comsol-server '''COMSOL Server'''] product offers an alternative
albeit more narrow means of access.

All users, including '''''non-Argonne users''''' may be able to run, over a web interface in their browser,
pre-parameterized "fixed-model" simulations that were made with the '''COMSOL Application Builder''' (requires Windows).

To inquire about using COMSOL server applications,
please contact us.

We will review possible purchase paths for COMSOL Server, and the ''considerable effort'' likely required by users and the CNM to develop and deploy a COMSOL application.

== Support ==
* For questions on how to run or start COMSOL:
*: Submit an [[HPC/Support | '''IT support request''']]
* For questions related to most COMSOL data or scripts, contact COMSOL support at:
*: [mailto:support@comsol.com '''support'''@comsol.com]
* '''Exception:''' If your support question might contain data that is subject to '''US Export Control''' regulations, contact instead:
*: [mailto:us-export-control@us.comsol.com '''us-export-control'''@'''us'''.comsol.com]

== The modes to run COMSOL ==
There are several ways of running COMSOL.

=== Running the desktop application ===
To edit models and analyze results, use the COMSOL graphical user interface (GUI). You can run it in several ways, each explained in more detail in the following sections.

# Run on Carbon, display on your desktop using the [http://en.wikipedia.org/wiki/X_Window_System X Window system (X11)].
#* Requires an ''X11 Server'' application running on your client machine, and SSH X11 forwarding.
#* The X11 protocol can be slow over LAN.
# Run on Carbon within a [[HPC/Network Access/Virtual Desktop | virtual desktop environment (VNC)]].
#* Requires a VNC client on your machine.
#* It can be confusing where VNC and the app runs.
#* The virtual desktop presents a (deliberatly) sparse windowing environment – Copy&Paste can be difficult to use.
# Run on your Argonne-provided computer, checking out Carbon's license remotely.
#* Requires a full installation of COMSOL on your desktoo or laptop computer.
#* You will require access to the COMSOL license server running on Carbon, either through VPN (fairly easy) or SSH tunneling (can be intimidating for first-time users). See below for details.

=== Running a batch calculation ===
When the computational stage of models takes longer than a few minutes to complete, it will be beneficial to submit save the model file and run it in parallel on Carbon.
See the [[#Batch jobs|section below]].

== Use with X11 ==
For casual use, and with [[HPC/Network Access/SSH Tunnel Setup on Linux and MacOS | X11 tunneling to Carbon]] set up, simply add to your <code>~/.bashrc</code> file on Carbon:
module load comsol

Then, from the command line on a Carbon login node, start the COMSOL GUI:
comsol &

License configuration is automatic on Carbon.

Comsol will use a number of cores to run calculations multithreaded by default. (On Carbon, COMSOL_NUM_THREADS is set.)

'''Quit the application''' as soon as you are done. Only one user can use Comsol at a time.

== Use with VNC ==
See general [[HPC/Network Access/Virtual Desktop | instructions on using VNC]], and the COMSOL example therein.

== Native desktop version ==

To get the fastest response from the COMSOL GUI, run a native COMSOL version on your desktop machine or laptop, called ''client machine'' below.

Enter license details either in the COMSOL Setup configuration dialog.

== Native desktop version — License configuration ==
A client's COMSOL license needs to be configured depending on how the client reaches CNM network resources. Follow the appropriate section:

* [[#License configuration when inside CNM or on CNM VPN|License configuration when '''inside CNM''' or on '''CNM VPN''']]
: Requires that the user's machine is logically within the network of CNM/NST, either by a wired/auth WiFi connection in the building or by VPN under a CNM/NST profile.
* [[#License configuration when outside CNM – requires SSH Tunneling|License configuration when '''outside CNM''' – requires SSH Tunneling]]
: ''Non-NST staff'' usually have a VPN profile that places the machine logically [[#License configuration when outside CNM – requires SSH Tunneling|outside of NST's network.]]

== License configuration when inside CNM or on CNM VPN ==
=== Ensure correct DNS search domains ===
* Inspect the client machine's network configuration.
* Ensure that '''two''' entries are listed as DNS Search Domains for the current network profile:
nst.anl.gov
cnm.anl.gov
: (The order of entries does not matter for the purposes considered here.)
* If the client machine is ''wired to CNM networks'', the domains are normally provided correctly by DHCP.
* If the client machine is ''connected over VPN'', you may need edit the respective network profile.
:* On a Mac, find the VPN configuration details under System Preferences, Network, Advanced… . You may need to unlock the Network panel first before you can make changes under "Advanced…". It appears that those changes are being saved and will be activated automatically the next time that this VPN session gets started.
: [[Image:VNC_add_cnm_to_DNS_1.png|400px]] [[Image:VNC_add_cnm_to_DNS_2.png|400px]]
:* On Windows or Linux machines, take ''analogous'' steps, using the platform's own configuration tools.

* Exclude license manager components from client desktop installation. (Those are all handled server-side.)

=== License format ===
* Floating Network License (FNL), "Three-server redundancy"

=== License servers ===
* On client machines, while running the COMSOL installer or license configuration, specify license servers and ports as follows:
1718 clicense1
1718 clicense2
1718 clicense3
: '''Note:''' Use ''short host names only''. (Technical Reason: The license must be usable both on Carbon's internal network as well as on divisional networks).
1718 clicense1<strike>.cnm.anl.gov</strike>
1718 clicense2<strike>.cnm.anl.gov</strike>
1718 clicense3<strike>.cnm.anl.gov</strike>

'''Advanced:''' A client machine's COMSOL license file <code><COMSOLDIR>/Multiphysics/license/license.dat</code> should look as follows:
<source lang="make">
SERVER clicense1 ANY 1718
SERVER clicense2 ANY 1718
SERVER clicense3 ANY 1718
USE_SERVER
</source>

<code><COMSOLDIR></code> is located by default at:
* '''Windows''': C:\Program files\COMSOL\COMSOL''nn''
* '''Linux''': /usr/local/comsol''nn''
* '''Mac''': /Applications/COMSOL''nn''
("''nn''" is the application's abbreviated version number.)

=== Network Troubleshooting ===
When using the license access model descibed here (as opposed to tunneled over SSH), ensure the following:
* Verify that short host names can be resolved to IP addresses. Either one of the following commands should produce sensible output:
host clicense1
: or:
ping -c 3 clicense1
* Verify that ''both license ports on the same server'' can be reached from your client machine:
telnet '''clicense1''' 1718
telnet '''clicense1''' 1719
: or:
telnet '''clicense2''' 1718
telnet '''clicense2''' 1719
: A ''successful'' connection looks like:
Trying 146.139.x.y...
Connected to clicense1.cnm.anl.gov.
Escape character is '^]'.
:: Do not enter any data. Instead, close each connection by hitting <code>Ctrl</code><code>]</code> (closing square bracket), followed by <code>Ctrl</code><code>D</code>.
: A ''failed'' connection gives one of these errors:
telnet: connect to address 146.139.x.y: Operation timed out
: or:
telnet: connect to address 146.139.x.y: Connection refused
:: In these cases, ask that your IP firewall configuration be reviewed and changed.

=== License configuration when outside CNM – requires SSH Tunneling ===
If you're comfortable with ssh and tunneling, set up client licensing as follows:

==== Configure COMSOL ====
The following needs to be done just once, typically during COMSOL installation:

In the COMSOL Installer, or in COMSOL Preferences, enter:
* '''License format:''' Floating Network License (FNL), "Port number and Hostname"
:: "Three-server redundancy" cannot be used because is cannot (easily) be accessed over SSH. Configure with just one server, and in rare tweak SSH to tunnel to the other server.
* '''Server:''' <code>127.0.0.1</code>
:: Do not use "localhost", as COMSOL interprets this incorrectly to mean that your machine ''is'' a license server.

* '''Port:''' <code>1718</code>


==== Provide server connection ====
This step needs to be done prior to running COMSOL in your desktop session:
* Connect to <code>mega</code> with the following ssh command:
ssh -L 1718:'''clicense1''':1718 -L 1719:'''clicense1''':1719 mega.cnm.anl.gov
* Leave the command running between invocations of COMSOL.

==== Run COMSOL ====
* Ensure your connection to "mega" is still running. If not, restart it.
* Start the COMSOL application.

** If you get an error that the number of licenses is "exhausted", that is confirmation that you correctly reached the license server, though other users are using licenses.
** If the license access attempt takes a long time or hangs, the service might temporarily run on a secondary server. In this case, stop the mega session and restart it, connecting to the alternate license server:
ssh -L 1718:clicense'''2''':1718 -L 1719:clicense'''2''':1719 mega.cnm.anl.gov

== License Troubleshooting ==
The following are typical error messages.
=== Could not obtain license for COMSOL Multiphysics GUI ===
* Error details:
License error: -4.
'''Licensed number of users already reached.'''
Feature: COMSOLGUI
License path: /Users/stern/Applications/COMSOL52a/license/license.dat:
FlexNet Licensing error:-4,132
* Reason: The connection to the license server succeeded, but the license token(s) you requested are in use.
* Solution: Wait a bit, or [http://carbonmgmt/ganglia/lic.php identify who's using COMSOL] and politely ask the user when they might be finished with their work.

=== License server machine is down or not responding ===
* Error details:
License error: -96.
See the system administrator about starting the license server system, or
make sure you're referring to the right host (see LM_LICENSE_FILE).
Feature: SERIAL
Hostname: clicense3
License path: /Users/stern/Applications/COMSOL52a/license/license.dat:
FlexNet Licensing error:-96,7
* Reason: The license servers cannot be reached at all. The <code>ping</code> or <code>host</code> tests mentioned above fail.
* Solution: Review the client license file and DNS lookup domains.

=== License file does not support this version ===
* Error details:
Feature: SERIAL
'''Application version > License version: 5.21 > 3.5'''
License path: /Users/stern/Applications/COMSOL52a/license/license.dat:
FlexNet Licensing error:-21,126
* Reason: Only the first of two steps for connecting to the license servers succeeds, which may happen when the redundant license service runs on the alternate server instead of the one you're connected to.
* Solution: When using SSH, try <code>clicense2</code> instead of <code>clicense1</code>, or vice versa. Revisit your <code>~/.ssh/config</code> file or use the alternate command line as shown in the SSH section above.

== Multiple CPU cores ==
Ways to start multi-core (from http://www.comsol.com/support/knowledgebase/1096/)
comsol -np 8

export COMSOL_NUM_THREADS=8 # automatically provided on ''Carbon''
comsol
* COMSOL does not benefit from hyperthreading (oversubscribing the physical core count).

== Batch jobs ==

* http://www.comsol.com/support/knowledgebase/1001/ .
* [[Media:HPC 2012-05-16 Comsol parallel usage.pdf]] (Modified from http://www.comsol.com/support/knowledgebase/1001/files/4027/cluster_install_linux_42a.pptx – Appendix slide, PBS)
* '''Note:''' MPD is not supported on Carbon (I patched the main comsol script), and no longer necessary at all in Comsol-4.3.

=== Sample Job file ===
See:
$COMSOL_HOME/sample.job

== Important COMSOL Support Articles ==
* Toplevel suport page: https://www.comsol.com/support

=== KB Articles ===
* [https://www.comsol.com/support/knowledgebase/1001 Running COMSOL® in parallel on clusters]
* [https://www.comsol.com/support/knowledgebase/1250 Running parametric sweeps, batch sweeps, and cluster sweeps from the command line]
* [https://www.comsol.com/support/knowledgebase/1255 Reducing the amount of solution data stored in a model]
* [https://www.comsol.com/support/knowledgebase/830 Handling large models]

=== Blog-style articles ===
* [https://www.comsol.com/blogs/the-power-of-the-batch-sweep/ The Power of the Batch Sweep]
* [https://www.comsol.com/blogs/how-to-run-simulations-in-batch-mode-from-the-command-line/ How to Run Simulations in Batch Mode from the Command Line]

HPC/Submitting and Managing Jobs

2026-02-17T20:45:29Z

Stern: /* Inspecting stuck jobs */

[[Image:HPC job flow.png|right|400px|thumb|Processing flow of a typical job.]]
__TOC__
== Introduction ==
To accept and run jobs, Carbon uses the [http://en.wikipedia.org/wiki/TORQUE_Resource_Manager Torque Resource Manager],
which is a variant of [http://en.wikipedia.org/wiki/Portable_Batch_System PBS].
A separate software component called a [http://en.wikipedia.org/wiki/Job_scheduler ''scheduler''] chooses the nodes and the start time for jobs according to the job parameters, resource availability, and configuration. Carbon uses the [http://www.adaptivecomputing.com/resources/docs/mwm/6-1-9/help.htm Moab Workload Manager] as scheduler. The resources used by each job (primarily the CPU time) are tracked and managed by the
[http://www.adaptivecomputing.com/resources/docs/gold/index.php Gold Allocation Manager].

=== Job processing ===
A job consists of a [[#Job Script|job script]] and applications called therein. The figure at the top shows schematically how a typical job is processed.
Unless otherwise told, Torque will execute the job script on your behalf on the first node allocated by Moab,
in your <code>$HOME</code> directory, and typically using your login shell. It is the ''script's responsibility'' to:
* Change to a suitable working directory.
* Recognize and use the other nodes of the job.
** This is generally done by directing an [http://en.wikipedia.org/wiki/Message_Passing_Interface MPI] program to start on the nodes listed in the job's <code>$PBS_NODEFILE</code>.
** A job is perfectly free to run more than one program, such as several MPI programs in sequence.
* Set or propagate environment variables as needed to other nodes.

=== Environment variables ===
The handling of environment variables across the cores and nodes that a job uses varies with the capabilities and configuration choices of the various MPI implementations.


* Variables usually are '''propagated core-to-core''' for all processes on the first node.
* Variables usually are '''not propagated node-to-node.'''
; Consequences:
* You must [[HPC/Software/Module Setup | '''configure modules and $PATH''']] in your <code>$HOME/.bashrc</code> file to help mpirun and other mechanisms locate your programs on each compute node. Otherwise mpirun will fail as soon as more than one node is involved, either by not finding the binary itself or some of its required shared libraries, or more insidiously, using different shared libraries.
* Both the variable <code>$PBS_NODEFILE</code> and the file it points to are only available on the first node.
* To propagate (export) environment variables, use mpirun/mpiexec. However, there is no standard or quasi-standard across MPI flavors.
** [http://www.open-mpi.org/doc/v1.4/man1/mpirun.1.php#sect17 OpenMPI's mpirun] selectively exports some variables and supports the <code>-x</code> flag to export additional variables.
** Intel's mpiexec/mpiexec.hydra offer <code>-envlist ''NAME,NAME,...''</code> instead. See Sec. 2.4 in the ''Reference Manual'' at [http://software.intel.com/en-us/articles/intel-mpi-library-documentation/].

See also: [[HPC/Directories|directory layout]].

== Submitting jobs ==
=== Job Script ===
A job is described to Torque by a ''job script,'' which in most cases is a shell script containing directives for Torque.
* [[HPC/Submitting and Managing Jobs/Example Job Script | '''Example Job Script''']].

=== Submission ===
To submit a job, use Torque's [http://www.adaptivecomputing.com/resources/docs/torque/4-1-4/Content/topics/commands/qsub.htm qsub] command on a login node.
qsub [-A accountname] [options] ''jobfile''
To review qsub operation and options, consult the following:
man qsub
qsub --help

; Advanced usage:
* You ''can'' submit follow-up jobs from within a parent job script while it runs on a compute node, but it may be better to leverage [http://www.adaptivecomputing.com/resources/docs/torque/4-1-4/Content/topics/commands/qsub.htm#W job dependencies.]
* [http://www.adaptivecomputing.com/resources/docs/torque/4-1-4/Content/topics/2-jobs/requestingRes.htm Requesting Resources]
* Alternatively, use Moab's [http://www.adaptivecomputing.com/resources/docs/mwm/6-1-9/Content/commands/msub.html msub] command.

=== Queues ===
See [[HPC/Submitting and Managing Jobs/Queues and Policies | Queues and Policies]].

== Checking job status ==
Query your jobs using commands similar to those on other HPC clusters.
You can use either [http://www.adaptivecomputing.com/resources/docs/torque/4-1-4/Content/topics/12-appendices/commandsOverview.htm Torque commands]
or [http://www.adaptivecomputing.com/resources/docs/mwm/6-1-9/Content/a.gcommandoverview.html Moab commands].
Both access the same information, but through different routes, and offer different display styles.

* To get the regular status output:
[http://www.adaptivecomputing.com/resources/docs/torque/4-1-4/Content/topics/commands/qstat.htm qstat] [-u $USER]
[http://www.adaptivecomputing.com/resources/docs/mwm/6-1-9/Content/commands/showq.html showq] [-u $USER]
* To get an ''alternate'' format (e.g. showing job names):
qstat -a [-u $USER]
showq -n [-u $USER]
: Option <code>-u $USER</code> shows only your jobs. You can literally type <code>$USER</code> instead of your user name.
* To get ''full'' job status information such as submit arguments and run directories, and do not wrap long lines ("-1" is the digit "one"):
qstat -f [''jobnum''] [-1]

== Inspecting stuck jobs ==
To diagnose problems with "stuck" jobs, use <code>qstat -s</code>, or simply Moab's
[http://docs.adaptivecomputing.com/9-1-3/suite/help.htm#topics/moabWorkloadManager/moabCommands/commands/checkjob.html checkjob] command. The last few lines of output give ''past events'' of the job and its current status.
qstat -s ''jobnum''
checkjob -v ''jobnum''

To get estimates on when your job might start:
showstart ''jobnum''

== Inspecting running jobs ==

=== Peeking at job output files ===
Each job has associated stdin and stderr streams which are gathered on its first node and are copied back only at job end time, according to the <code>#PBS</code> directives -e, -o, or -j [eo|oe]. If the user does explicitly capture any output in the script (using shell redirections <code>">"</code>) it can be difficult to assess a job's progress.

To inspect these streams ''while the job is running'', and also to review a job's script, use the <code>qpeek</code> command:
qpeek ''jobnum''
qpeek -eo ''jobnum''
qpeek -s ''jobnum''
Read its manual page for full documentation:
qpeek --man
This command is a Carbon re-implementation of a Torque "contrib" script.

To inspect a job's processes running on its compute nodes:
qpeek -p ''jobnum''

=== Which nodes does a running job use? ===
qstat -n [-1] ''jobnum''

== Changing job resources after submission ==

Use <code>qalter</code> to modify attributes of a job.
Only attributes specified on the command line will be changed.
If any of the specified attributes cannot be modified for any reason, ''none'' of that job's attributes will be modified.
qalter [-l ''resource''] […] ''jobnumber'' …
qalter -A ''accountspec'' ''jobnumber'' …
* Most attributes can be changed only ''before'' the job starts. Others, notably
 mail options and  job dependencies can be meaningfully adjusted after start.

; Advanced usage:
* ''Extending'' the <code>walltime</code> resource of an already running job requires Torque administrator permission. You yourself can extend prior to running, and shorten always. '''Warning:''' In either case, the new <code>walltime</code> limit will be measured from start, not as remaining time. If a new walltime limit is shorter than the elapsed time, the job will be killed right away.
* You cannot modify the job script itself after submission. You will have to remove and resubmit the job.
Contact me ([[User:Stern|stern]]) to arrange for interventions.

== Canceling (removing) jobs ==
To retract a queued job or kill an already running job:
qdel ''jobnumber''

Torque epilogue scripts will immediately remove remaining user processes and user files from the used compute nodes upon job termination, be it natural or after qdel.

== Declaring that a job requires licenses ==
A job that is to run software that requires a commercial license must declare such to the job scheduler (software that decides when and on which nodes the job could start).
The job scheduler monitors known licenses for availability, and so will know which and how many licenses are available at any given time.

'''Note:''' License availability checks do not happen instantaneously but rather by polling, roughly once every minute.
: This means that if you or another user happen to start a license-consuming application after the last check but before the relevant newly started job tries to access the same license, that job will terminate with a license failure.

=== One license type ===
To request a single license token of a single type, use a <code>gres</code> ("general resource") directive:
#PBS -l '''gres'''=''license_name''
To request several tokens of a type, place either a <code>:</code> or <code>+</code> as separator, followed by the number of tokens required:
#PBS -l '''gres'''=''license_name:count''

For compatibility, the resource type <code>software</code> is accepted, but ''not for multiple types'' of licenses.
#PBS -l '''software'''=''license_name''
#or
#PBS -l '''software'''=''license_name:count''

=== Several license types ===
To request several different licenses, use <code>gres</code> with list of license tokens and their optional counts separated by <code>'''%'''</code>:
#PBS -l gres=''license1_name'''''%'''''license2_name''
#PBS -l gres=''license1_name:count1'''''%'''''license2_name''

The <code>software</code> resource ''cannot'' request multiple license types.
(If several <code>software</code> resources are requested, only the ''last one'' would be active.)

== Interactive node access ==
There are two ways to interactively access nodes:
* Requesting an interactive job
qsub -I [-l nodes=…]
: This is a special Torque job submission that will wait for the requested nodes to become available, and then you will be dropped into a login shell on the first core of the first node. $PBS_NODEFILE is set as in regular (non-interactive) jobs.
* Side-access a node that already runs a job, such as to debug or inspect running processes or node-local files.
: You can use <code>ssh</code> to interactively access any compute node on which a job of yours is running (use <code>qsub -n ''jobnum''</code> to find out the node names). As soon as a node no longer runs at least ''one'' of your jobs, your ssh session to that node will be terminated. You may find the following commands helpful:
/usr/sbin/lsof -u $USER
/usr/sbin/lsof -c ''command_name''
/usr/sbin/lsof -p ''pid''
: - '''l'''i'''s'''t '''o'''pen '''f'''iles; [http://linux.die.net/man/8/lsof man page.]
ps -fu $USER
: - list your own processes.
psuser
: - a Carbon-specific wrapper for [http://linux.die.net/man/1/ps ps] to list user-only processes (yours and other regular users', excluding system processes).

== Job notification mails ==
You can request a notification message to be sent to you when a job begins, ends, or aborts. The message is rather terse and looks like this:
'''Subject:''' PBS Job 281399.sched1.carboncluster

PBS Job Id: 281399.sched1.carboncluster
Job Name: test.job
Exec host: n103/0
Execution terminated
Exit_status=0
resources_used.cput=00:03:00
resources_used.mem=123kb
resources_used.vmem=240kb
resources_used.walltime=00:03:04

In the above, Exit_status=0 reflects the shell exit status, where 0 means success.

The following means your job was terminated because it ran beyond its allocated walltime:
Exit_status=271
: The shell exit status is this number mod 256, in this case <code>271 % 256 = 15</code>, corresonding to <code>SIGTERM</code>.

== Recording where a job ran ==
The [[#Job notification mails]] explained above do not include $PBS_O_WORKDIR. You will have to arrange for yourself to record that information.
One way to do that would be to simply record all job numbers and their working directories at the beginning of each job:
<source lang="bash">
#!/bin/bash
#PBS -l nodes=…
#PBS -l walltime=…
#PBS -m bea

cd $PBS_O_WORKDIR
echo $PBS_JOBID $PWD >> $HOME/jobdirs

…
</source>
To query a job's directory, run:
grep ''jobnumber'' ~/jobdirs

== Sending mail while a job is running ==
It may be useful to have a job automatically "phone home" sometime after it started.
This can be accomplished at job submission time by arranging a mail to be sent, using a few added commands in the job script.

<syntaxhighlight lang="bash">
#!/bin/bash
#PBS …

cd $PBS_O_WORKDIR

# arrange notification mail
(
sleep 600 # time offset parameter, in seconds
tail -20 somefile.out | mailx -s "$PBS_JOBID: partial log" $USER
) &

# main work here
…
</syntaxhighlight>

Explanation:
* The commands inside <code>( … ) &</code> run in a background track aside the main work.
* The first command, <code>sleep</code>, delays further action (in this track only) by the specified time in seconds.
** To emphasize minutes, write <code>sleep $(( 10 * 60 ))</code> .
** To set the time relative to the walltime limit, leverage the <code>$PBS_WALLTIME</code> variable, e.g. as in <code>sleep $(( PBS_WALLTIME - ( ''minutes'' * 60 ) )).</code>
* The <code>tail</code> command is a stand-in for any inspection command you might want to run. An even simpler command could be <code>ls -lt</code>. You could also use several commands enclosed by parentheses, <code>( cmd1; cmd2; ... )</code>, to construct a more elaborate mail body.
* The <code>mailx</code> command, following the pipe symbol <code>|</code>, reads the output of your inspection command and sends you a mail with the subject specified by the <code>-s</code> option.
* If the main part of the job finishes before the pre-mail sleep is over no mail will be sent.

Note: The Moab scheduler also supports [http://www.adaptivecomputing.com/resources/docs/mwm/6-1-9/Content/topics/triggers/triggersIndex.html triggers], but they run in a different environment, not on an allocated node.

[[Category:HPC|Getting Started]]

HPC/Submitting and Managing Jobs

2026-02-17T20:44:42Z

Stern: /* Inspecting stuck jobs */

[[Image:HPC job flow.png|right|400px|thumb|Processing flow of a typical job.]]
__TOC__
== Introduction ==
To accept and run jobs, Carbon uses the [http://en.wikipedia.org/wiki/TORQUE_Resource_Manager Torque Resource Manager],
which is a variant of [http://en.wikipedia.org/wiki/Portable_Batch_System PBS].
A separate software component called a [http://en.wikipedia.org/wiki/Job_scheduler ''scheduler''] chooses the nodes and the start time for jobs according to the job parameters, resource availability, and configuration. Carbon uses the [http://www.adaptivecomputing.com/resources/docs/mwm/6-1-9/help.htm Moab Workload Manager] as scheduler. The resources used by each job (primarily the CPU time) are tracked and managed by the
[http://www.adaptivecomputing.com/resources/docs/gold/index.php Gold Allocation Manager].

=== Job processing ===
A job consists of a [[#Job Script|job script]] and applications called therein. The figure at the top shows schematically how a typical job is processed.
Unless otherwise told, Torque will execute the job script on your behalf on the first node allocated by Moab,
in your <code>$HOME</code> directory, and typically using your login shell. It is the ''script's responsibility'' to:
* Change to a suitable working directory.
* Recognize and use the other nodes of the job.
** This is generally done by directing an [http://en.wikipedia.org/wiki/Message_Passing_Interface MPI] program to start on the nodes listed in the job's <code>$PBS_NODEFILE</code>.
** A job is perfectly free to run more than one program, such as several MPI programs in sequence.
* Set or propagate environment variables as needed to other nodes.

=== Environment variables ===
The handling of environment variables across the cores and nodes that a job uses varies with the capabilities and configuration choices of the various MPI implementations.


* Variables usually are '''propagated core-to-core''' for all processes on the first node.
* Variables usually are '''not propagated node-to-node.'''
; Consequences:
* You must [[HPC/Software/Module Setup | '''configure modules and $PATH''']] in your <code>$HOME/.bashrc</code> file to help mpirun and other mechanisms locate your programs on each compute node. Otherwise mpirun will fail as soon as more than one node is involved, either by not finding the binary itself or some of its required shared libraries, or more insidiously, using different shared libraries.
* Both the variable <code>$PBS_NODEFILE</code> and the file it points to are only available on the first node.
* To propagate (export) environment variables, use mpirun/mpiexec. However, there is no standard or quasi-standard across MPI flavors.
** [http://www.open-mpi.org/doc/v1.4/man1/mpirun.1.php#sect17 OpenMPI's mpirun] selectively exports some variables and supports the <code>-x</code> flag to export additional variables.
** Intel's mpiexec/mpiexec.hydra offer <code>-envlist ''NAME,NAME,...''</code> instead. See Sec. 2.4 in the ''Reference Manual'' at [http://software.intel.com/en-us/articles/intel-mpi-library-documentation/].

See also: [[HPC/Directories|directory layout]].

== Submitting jobs ==
=== Job Script ===
A job is described to Torque by a ''job script,'' which in most cases is a shell script containing directives for Torque.
* [[HPC/Submitting and Managing Jobs/Example Job Script | '''Example Job Script''']].

=== Submission ===
To submit a job, use Torque's [http://www.adaptivecomputing.com/resources/docs/torque/4-1-4/Content/topics/commands/qsub.htm qsub] command on a login node.
qsub [-A accountname] [options] ''jobfile''
To review qsub operation and options, consult the following:
man qsub
qsub --help

; Advanced usage:
* You ''can'' submit follow-up jobs from within a parent job script while it runs on a compute node, but it may be better to leverage [http://www.adaptivecomputing.com/resources/docs/torque/4-1-4/Content/topics/commands/qsub.htm#W job dependencies.]
* [http://www.adaptivecomputing.com/resources/docs/torque/4-1-4/Content/topics/2-jobs/requestingRes.htm Requesting Resources]
* Alternatively, use Moab's [http://www.adaptivecomputing.com/resources/docs/mwm/6-1-9/Content/commands/msub.html msub] command.

=== Queues ===
See [[HPC/Submitting and Managing Jobs/Queues and Policies | Queues and Policies]].

== Checking job status ==
Query your jobs using commands similar to those on other HPC clusters.
You can use either [http://www.adaptivecomputing.com/resources/docs/torque/4-1-4/Content/topics/12-appendices/commandsOverview.htm Torque commands]
or [http://www.adaptivecomputing.com/resources/docs/mwm/6-1-9/Content/a.gcommandoverview.html Moab commands].
Both access the same information, but through different routes, and offer different display styles.

* To get the regular status output:
[http://www.adaptivecomputing.com/resources/docs/torque/4-1-4/Content/topics/commands/qstat.htm qstat] [-u $USER]
[http://www.adaptivecomputing.com/resources/docs/mwm/6-1-9/Content/commands/showq.html showq] [-u $USER]
* To get an ''alternate'' format (e.g. showing job names):
qstat -a [-u $USER]
showq -n [-u $USER]
: Option <code>-u $USER</code> shows only your jobs. You can literally type <code>$USER</code> instead of your user name.
* To get ''full'' job status information such as submit arguments and run directories, and do not wrap long lines ("-1" is the digit "one"):
qstat -f [''jobnum''] [-1]

== Inspecting stuck jobs ==
To diagnose problems with "stuck" jobs, use <code>qstat -s</code>, or simply Moab's
[http://docs.adaptivecomputing.com/9-1-3/suite/help.htm#topics/moabWorkloadManager/moabCommands/commands/checkjob.html checkjob] command. The last few lines of output give ''past events'' of the job and its current status.
qstat -s ''jobnum''
checkjob -v ''jobnum''

To get estimates on when your job might start:
showstart [-e all] ''jobnum''

== Inspecting running jobs ==

=== Peeking at job output files ===
Each job has associated stdin and stderr streams which are gathered on its first node and are copied back only at job end time, according to the <code>#PBS</code> directives -e, -o, or -j [eo|oe]. If the user does explicitly capture any output in the script (using shell redirections <code>">"</code>) it can be difficult to assess a job's progress.

To inspect these streams ''while the job is running'', and also to review a job's script, use the <code>qpeek</code> command:
qpeek ''jobnum''
qpeek -eo ''jobnum''
qpeek -s ''jobnum''
Read its manual page for full documentation:
qpeek --man
This command is a Carbon re-implementation of a Torque "contrib" script.

To inspect a job's processes running on its compute nodes:
qpeek -p ''jobnum''

=== Which nodes does a running job use? ===
qstat -n [-1] ''jobnum''

== Changing job resources after submission ==

Use <code>qalter</code> to modify attributes of a job.
Only attributes specified on the command line will be changed.
If any of the specified attributes cannot be modified for any reason, ''none'' of that job's attributes will be modified.
qalter [-l ''resource''] […] ''jobnumber'' …
qalter -A ''accountspec'' ''jobnumber'' …
* Most attributes can be changed only ''before'' the job starts. Others, notably
 mail options and  job dependencies can be meaningfully adjusted after start.

; Advanced usage:
* ''Extending'' the <code>walltime</code> resource of an already running job requires Torque administrator permission. You yourself can extend prior to running, and shorten always. '''Warning:''' In either case, the new <code>walltime</code> limit will be measured from start, not as remaining time. If a new walltime limit is shorter than the elapsed time, the job will be killed right away.
* You cannot modify the job script itself after submission. You will have to remove and resubmit the job.
Contact me ([[User:Stern|stern]]) to arrange for interventions.

== Canceling (removing) jobs ==
To retract a queued job or kill an already running job:
qdel ''jobnumber''

Torque epilogue scripts will immediately remove remaining user processes and user files from the used compute nodes upon job termination, be it natural or after qdel.

== Declaring that a job requires licenses ==
A job that is to run software that requires a commercial license must declare such to the job scheduler (software that decides when and on which nodes the job could start).
The job scheduler monitors known licenses for availability, and so will know which and how many licenses are available at any given time.

'''Note:''' License availability checks do not happen instantaneously but rather by polling, roughly once every minute.
: This means that if you or another user happen to start a license-consuming application after the last check but before the relevant newly started job tries to access the same license, that job will terminate with a license failure.

=== One license type ===
To request a single license token of a single type, use a <code>gres</code> ("general resource") directive:
#PBS -l '''gres'''=''license_name''
To request several tokens of a type, place either a <code>:</code> or <code>+</code> as separator, followed by the number of tokens required:
#PBS -l '''gres'''=''license_name:count''

For compatibility, the resource type <code>software</code> is accepted, but ''not for multiple types'' of licenses.
#PBS -l '''software'''=''license_name''
#or
#PBS -l '''software'''=''license_name:count''

=== Several license types ===
To request several different licenses, use <code>gres</code> with list of license tokens and their optional counts separated by <code>'''%'''</code>:
#PBS -l gres=''license1_name'''''%'''''license2_name''
#PBS -l gres=''license1_name:count1'''''%'''''license2_name''

The <code>software</code> resource ''cannot'' request multiple license types.
(If several <code>software</code> resources are requested, only the ''last one'' would be active.)

== Interactive node access ==
There are two ways to interactively access nodes:
* Requesting an interactive job
qsub -I [-l nodes=…]
: This is a special Torque job submission that will wait for the requested nodes to become available, and then you will be dropped into a login shell on the first core of the first node. $PBS_NODEFILE is set as in regular (non-interactive) jobs.
* Side-access a node that already runs a job, such as to debug or inspect running processes or node-local files.
: You can use <code>ssh</code> to interactively access any compute node on which a job of yours is running (use <code>qsub -n ''jobnum''</code> to find out the node names). As soon as a node no longer runs at least ''one'' of your jobs, your ssh session to that node will be terminated. You may find the following commands helpful:
/usr/sbin/lsof -u $USER
/usr/sbin/lsof -c ''command_name''
/usr/sbin/lsof -p ''pid''
: - '''l'''i'''s'''t '''o'''pen '''f'''iles; [http://linux.die.net/man/8/lsof man page.]
ps -fu $USER
: - list your own processes.
psuser
: - a Carbon-specific wrapper for [http://linux.die.net/man/1/ps ps] to list user-only processes (yours and other regular users', excluding system processes).

== Job notification mails ==
You can request a notification message to be sent to you when a job begins, ends, or aborts. The message is rather terse and looks like this:
'''Subject:''' PBS Job 281399.sched1.carboncluster

PBS Job Id: 281399.sched1.carboncluster
Job Name: test.job
Exec host: n103/0
Execution terminated
Exit_status=0
resources_used.cput=00:03:00
resources_used.mem=123kb
resources_used.vmem=240kb
resources_used.walltime=00:03:04

In the above, Exit_status=0 reflects the shell exit status, where 0 means success.

The following means your job was terminated because it ran beyond its allocated walltime:
Exit_status=271
: The shell exit status is this number mod 256, in this case <code>271 % 256 = 15</code>, corresonding to <code>SIGTERM</code>.

== Recording where a job ran ==
The [[#Job notification mails]] explained above do not include $PBS_O_WORKDIR. You will have to arrange for yourself to record that information.
One way to do that would be to simply record all job numbers and their working directories at the beginning of each job:
<source lang="bash">
#!/bin/bash
#PBS -l nodes=…
#PBS -l walltime=…
#PBS -m bea

cd $PBS_O_WORKDIR
echo $PBS_JOBID $PWD >> $HOME/jobdirs

…
</source>
To query a job's directory, run:
grep ''jobnumber'' ~/jobdirs

== Sending mail while a job is running ==
It may be useful to have a job automatically "phone home" sometime after it started.
This can be accomplished at job submission time by arranging a mail to be sent, using a few added commands in the job script.

<syntaxhighlight lang="bash">
#!/bin/bash
#PBS …

cd $PBS_O_WORKDIR

# arrange notification mail
(
sleep 600 # time offset parameter, in seconds
tail -20 somefile.out | mailx -s "$PBS_JOBID: partial log" $USER
) &

# main work here
…
</syntaxhighlight>

Explanation:
* The commands inside <code>( … ) &</code> run in a background track aside the main work.
* The first command, <code>sleep</code>, delays further action (in this track only) by the specified time in seconds.
** To emphasize minutes, write <code>sleep $(( 10 * 60 ))</code> .
** To set the time relative to the walltime limit, leverage the <code>$PBS_WALLTIME</code> variable, e.g. as in <code>sleep $(( PBS_WALLTIME - ( ''minutes'' * 60 ) )).</code>
* The <code>tail</code> command is a stand-in for any inspection command you might want to run. An even simpler command could be <code>ls -lt</code>. You could also use several commands enclosed by parentheses, <code>( cmd1; cmd2; ... )</code>, to construct a more elaborate mail body.
* The <code>mailx</code> command, following the pipe symbol <code>|</code>, reads the output of your inspection command and sends you a mail with the subject specified by the <code>-s</code> option.
* If the main part of the job finishes before the pre-mail sleep is over no mail will be sent.

Note: The Moab scheduler also supports [http://www.adaptivecomputing.com/resources/docs/mwm/6-1-9/Content/topics/triggers/triggersIndex.html triggers], but they run in a different environment, not on an allocated node.

[[Category:HPC|Getting Started]]

HPC/Submitting and Managing Jobs

2026-02-17T20:44:12Z

Stern: /* Inspecting stuck jobs */

[[Image:HPC job flow.png|right|400px|thumb|Processing flow of a typical job.]]
__TOC__
== Introduction ==
To accept and run jobs, Carbon uses the [http://en.wikipedia.org/wiki/TORQUE_Resource_Manager Torque Resource Manager],
which is a variant of [http://en.wikipedia.org/wiki/Portable_Batch_System PBS].
A separate software component called a [http://en.wikipedia.org/wiki/Job_scheduler ''scheduler''] chooses the nodes and the start time for jobs according to the job parameters, resource availability, and configuration. Carbon uses the [http://www.adaptivecomputing.com/resources/docs/mwm/6-1-9/help.htm Moab Workload Manager] as scheduler. The resources used by each job (primarily the CPU time) are tracked and managed by the
[http://www.adaptivecomputing.com/resources/docs/gold/index.php Gold Allocation Manager].

=== Job processing ===
A job consists of a [[#Job Script|job script]] and applications called therein. The figure at the top shows schematically how a typical job is processed.
Unless otherwise told, Torque will execute the job script on your behalf on the first node allocated by Moab,
in your <code>$HOME</code> directory, and typically using your login shell. It is the ''script's responsibility'' to:
* Change to a suitable working directory.
* Recognize and use the other nodes of the job.
** This is generally done by directing an [http://en.wikipedia.org/wiki/Message_Passing_Interface MPI] program to start on the nodes listed in the job's <code>$PBS_NODEFILE</code>.
** A job is perfectly free to run more than one program, such as several MPI programs in sequence.
* Set or propagate environment variables as needed to other nodes.

=== Environment variables ===
The handling of environment variables across the cores and nodes that a job uses varies with the capabilities and configuration choices of the various MPI implementations.


* Variables usually are '''propagated core-to-core''' for all processes on the first node.
* Variables usually are '''not propagated node-to-node.'''
; Consequences:
* You must [[HPC/Software/Module Setup | '''configure modules and $PATH''']] in your <code>$HOME/.bashrc</code> file to help mpirun and other mechanisms locate your programs on each compute node. Otherwise mpirun will fail as soon as more than one node is involved, either by not finding the binary itself or some of its required shared libraries, or more insidiously, using different shared libraries.
* Both the variable <code>$PBS_NODEFILE</code> and the file it points to are only available on the first node.
* To propagate (export) environment variables, use mpirun/mpiexec. However, there is no standard or quasi-standard across MPI flavors.
** [http://www.open-mpi.org/doc/v1.4/man1/mpirun.1.php#sect17 OpenMPI's mpirun] selectively exports some variables and supports the <code>-x</code> flag to export additional variables.
** Intel's mpiexec/mpiexec.hydra offer <code>-envlist ''NAME,NAME,...''</code> instead. See Sec. 2.4 in the ''Reference Manual'' at [http://software.intel.com/en-us/articles/intel-mpi-library-documentation/].

See also: [[HPC/Directories|directory layout]].

== Submitting jobs ==
=== Job Script ===
A job is described to Torque by a ''job script,'' which in most cases is a shell script containing directives for Torque.
* [[HPC/Submitting and Managing Jobs/Example Job Script | '''Example Job Script''']].

=== Submission ===
To submit a job, use Torque's [http://www.adaptivecomputing.com/resources/docs/torque/4-1-4/Content/topics/commands/qsub.htm qsub] command on a login node.
qsub [-A accountname] [options] ''jobfile''
To review qsub operation and options, consult the following:
man qsub
qsub --help

; Advanced usage:
* You ''can'' submit follow-up jobs from within a parent job script while it runs on a compute node, but it may be better to leverage [http://www.adaptivecomputing.com/resources/docs/torque/4-1-4/Content/topics/commands/qsub.htm#W job dependencies.]
* [http://www.adaptivecomputing.com/resources/docs/torque/4-1-4/Content/topics/2-jobs/requestingRes.htm Requesting Resources]
* Alternatively, use Moab's [http://www.adaptivecomputing.com/resources/docs/mwm/6-1-9/Content/commands/msub.html msub] command.

=== Queues ===
See [[HPC/Submitting and Managing Jobs/Queues and Policies | Queues and Policies]].

== Checking job status ==
Query your jobs using commands similar to those on other HPC clusters.
You can use either [http://www.adaptivecomputing.com/resources/docs/torque/4-1-4/Content/topics/12-appendices/commandsOverview.htm Torque commands]
or [http://www.adaptivecomputing.com/resources/docs/mwm/6-1-9/Content/a.gcommandoverview.html Moab commands].
Both access the same information, but through different routes, and offer different display styles.

* To get the regular status output:
[http://www.adaptivecomputing.com/resources/docs/torque/4-1-4/Content/topics/commands/qstat.htm qstat] [-u $USER]
[http://www.adaptivecomputing.com/resources/docs/mwm/6-1-9/Content/commands/showq.html showq] [-u $USER]
* To get an ''alternate'' format (e.g. showing job names):
qstat -a [-u $USER]
showq -n [-u $USER]
: Option <code>-u $USER</code> shows only your jobs. You can literally type <code>$USER</code> instead of your user name.
* To get ''full'' job status information such as submit arguments and run directories, and do not wrap long lines ("-1" is the digit "one"):
qstat -f [''jobnum''] [-1]

== Inspecting stuck jobs ==
To diagnose problems with "stuck" jobs, use <code>qstat -s</code>, or simply Moab's
[http://docs.adaptivecomputing.com/9-1-3/suite/help.htm#topics/moabWorkloadManager/moabCommands/commands/checkjob.html?Highlight=checkjob checkjob] command. The last few lines of output give ''past events'' of the job and its current status.
qstat -s ''jobnum''
checkjob -v ''jobnum''

To get estimates on when your job might start:
showstart [-e all] ''jobnum''

== Inspecting running jobs ==

=== Peeking at job output files ===
Each job has associated stdin and stderr streams which are gathered on its first node and are copied back only at job end time, according to the <code>#PBS</code> directives -e, -o, or -j [eo|oe]. If the user does explicitly capture any output in the script (using shell redirections <code>">"</code>) it can be difficult to assess a job's progress.

To inspect these streams ''while the job is running'', and also to review a job's script, use the <code>qpeek</code> command:
qpeek ''jobnum''
qpeek -eo ''jobnum''
qpeek -s ''jobnum''
Read its manual page for full documentation:
qpeek --man
This command is a Carbon re-implementation of a Torque "contrib" script.

To inspect a job's processes running on its compute nodes:
qpeek -p ''jobnum''

=== Which nodes does a running job use? ===
qstat -n [-1] ''jobnum''

== Changing job resources after submission ==

Use <code>qalter</code> to modify attributes of a job.
Only attributes specified on the command line will be changed.
If any of the specified attributes cannot be modified for any reason, ''none'' of that job's attributes will be modified.
qalter [-l ''resource''] […] ''jobnumber'' …
qalter -A ''accountspec'' ''jobnumber'' …
* Most attributes can be changed only ''before'' the job starts. Others, notably
 mail options and  job dependencies can be meaningfully adjusted after start.

; Advanced usage:
* ''Extending'' the <code>walltime</code> resource of an already running job requires Torque administrator permission. You yourself can extend prior to running, and shorten always. '''Warning:''' In either case, the new <code>walltime</code> limit will be measured from start, not as remaining time. If a new walltime limit is shorter than the elapsed time, the job will be killed right away.
* You cannot modify the job script itself after submission. You will have to remove and resubmit the job.
Contact me ([[User:Stern|stern]]) to arrange for interventions.

== Canceling (removing) jobs ==
To retract a queued job or kill an already running job:
qdel ''jobnumber''

Torque epilogue scripts will immediately remove remaining user processes and user files from the used compute nodes upon job termination, be it natural or after qdel.

== Declaring that a job requires licenses ==
A job that is to run software that requires a commercial license must declare such to the job scheduler (software that decides when and on which nodes the job could start).
The job scheduler monitors known licenses for availability, and so will know which and how many licenses are available at any given time.

'''Note:''' License availability checks do not happen instantaneously but rather by polling, roughly once every minute.
: This means that if you or another user happen to start a license-consuming application after the last check but before the relevant newly started job tries to access the same license, that job will terminate with a license failure.

=== One license type ===
To request a single license token of a single type, use a <code>gres</code> ("general resource") directive:
#PBS -l '''gres'''=''license_name''
To request several tokens of a type, place either a <code>:</code> or <code>+</code> as separator, followed by the number of tokens required:
#PBS -l '''gres'''=''license_name:count''

For compatibility, the resource type <code>software</code> is accepted, but ''not for multiple types'' of licenses.
#PBS -l '''software'''=''license_name''
#or
#PBS -l '''software'''=''license_name:count''

=== Several license types ===
To request several different licenses, use <code>gres</code> with list of license tokens and their optional counts separated by <code>'''%'''</code>:
#PBS -l gres=''license1_name'''''%'''''license2_name''
#PBS -l gres=''license1_name:count1'''''%'''''license2_name''

The <code>software</code> resource ''cannot'' request multiple license types.
(If several <code>software</code> resources are requested, only the ''last one'' would be active.)

== Interactive node access ==
There are two ways to interactively access nodes:
* Requesting an interactive job
qsub -I [-l nodes=…]
: This is a special Torque job submission that will wait for the requested nodes to become available, and then you will be dropped into a login shell on the first core of the first node. $PBS_NODEFILE is set as in regular (non-interactive) jobs.
* Side-access a node that already runs a job, such as to debug or inspect running processes or node-local files.
: You can use <code>ssh</code> to interactively access any compute node on which a job of yours is running (use <code>qsub -n ''jobnum''</code> to find out the node names). As soon as a node no longer runs at least ''one'' of your jobs, your ssh session to that node will be terminated. You may find the following commands helpful:
/usr/sbin/lsof -u $USER
/usr/sbin/lsof -c ''command_name''
/usr/sbin/lsof -p ''pid''
: - '''l'''i'''s'''t '''o'''pen '''f'''iles; [http://linux.die.net/man/8/lsof man page.]
ps -fu $USER
: - list your own processes.
psuser
: - a Carbon-specific wrapper for [http://linux.die.net/man/1/ps ps] to list user-only processes (yours and other regular users', excluding system processes).

== Job notification mails ==
You can request a notification message to be sent to you when a job begins, ends, or aborts. The message is rather terse and looks like this:
'''Subject:''' PBS Job 281399.sched1.carboncluster

PBS Job Id: 281399.sched1.carboncluster
Job Name: test.job
Exec host: n103/0
Execution terminated
Exit_status=0
resources_used.cput=00:03:00
resources_used.mem=123kb
resources_used.vmem=240kb
resources_used.walltime=00:03:04

In the above, Exit_status=0 reflects the shell exit status, where 0 means success.

The following means your job was terminated because it ran beyond its allocated walltime:
Exit_status=271
: The shell exit status is this number mod 256, in this case <code>271 % 256 = 15</code>, corresonding to <code>SIGTERM</code>.

== Recording where a job ran ==
The [[#Job notification mails]] explained above do not include $PBS_O_WORKDIR. You will have to arrange for yourself to record that information.
One way to do that would be to simply record all job numbers and their working directories at the beginning of each job:
<source lang="bash">
#!/bin/bash
#PBS -l nodes=…
#PBS -l walltime=…
#PBS -m bea

cd $PBS_O_WORKDIR
echo $PBS_JOBID $PWD >> $HOME/jobdirs

…
</source>
To query a job's directory, run:
grep ''jobnumber'' ~/jobdirs

== Sending mail while a job is running ==
It may be useful to have a job automatically "phone home" sometime after it started.
This can be accomplished at job submission time by arranging a mail to be sent, using a few added commands in the job script.

<syntaxhighlight lang="bash">
#!/bin/bash
#PBS …

cd $PBS_O_WORKDIR

# arrange notification mail
(
sleep 600 # time offset parameter, in seconds
tail -20 somefile.out | mailx -s "$PBS_JOBID: partial log" $USER
) &

# main work here
…
</syntaxhighlight>

Explanation:
* The commands inside <code>( … ) &</code> run in a background track aside the main work.
* The first command, <code>sleep</code>, delays further action (in this track only) by the specified time in seconds.
** To emphasize minutes, write <code>sleep $(( 10 * 60 ))</code> .
** To set the time relative to the walltime limit, leverage the <code>$PBS_WALLTIME</code> variable, e.g. as in <code>sleep $(( PBS_WALLTIME - ( ''minutes'' * 60 ) )).</code>
* The <code>tail</code> command is a stand-in for any inspection command you might want to run. An even simpler command could be <code>ls -lt</code>. You could also use several commands enclosed by parentheses, <code>( cmd1; cmd2; ... )</code>, to construct a more elaborate mail body.
* The <code>mailx</code> command, following the pipe symbol <code>|</code>, reads the output of your inspection command and sends you a mail with the subject specified by the <code>-s</code> option.
* If the main part of the job finishes before the pre-mail sleep is over no mail will be sent.

Note: The Moab scheduler also supports [http://www.adaptivecomputing.com/resources/docs/mwm/6-1-9/Content/topics/triggers/triggersIndex.html triggers], but they run in a different environment, not on an allocated node.

[[Category:HPC|Getting Started]]

HPC/Submitting and Managing Jobs

2026-02-17T20:40:49Z

Stern: /* Inspecting "stuck" jobs */

[[Image:HPC job flow.png|right|400px|thumb|Processing flow of a typical job.]]
__TOC__
== Introduction ==
To accept and run jobs, Carbon uses the [http://en.wikipedia.org/wiki/TORQUE_Resource_Manager Torque Resource Manager],
which is a variant of [http://en.wikipedia.org/wiki/Portable_Batch_System PBS].
A separate software component called a [http://en.wikipedia.org/wiki/Job_scheduler ''scheduler''] chooses the nodes and the start time for jobs according to the job parameters, resource availability, and configuration. Carbon uses the [http://www.adaptivecomputing.com/resources/docs/mwm/6-1-9/help.htm Moab Workload Manager] as scheduler. The resources used by each job (primarily the CPU time) are tracked and managed by the
[http://www.adaptivecomputing.com/resources/docs/gold/index.php Gold Allocation Manager].

=== Job processing ===
A job consists of a [[#Job Script|job script]] and applications called therein. The figure at the top shows schematically how a typical job is processed.
Unless otherwise told, Torque will execute the job script on your behalf on the first node allocated by Moab,
in your <code>$HOME</code> directory, and typically using your login shell. It is the ''script's responsibility'' to:
* Change to a suitable working directory.
* Recognize and use the other nodes of the job.
** This is generally done by directing an [http://en.wikipedia.org/wiki/Message_Passing_Interface MPI] program to start on the nodes listed in the job's <code>$PBS_NODEFILE</code>.
** A job is perfectly free to run more than one program, such as several MPI programs in sequence.
* Set or propagate environment variables as needed to other nodes.

=== Environment variables ===
The handling of environment variables across the cores and nodes that a job uses varies with the capabilities and configuration choices of the various MPI implementations.


* Variables usually are '''propagated core-to-core''' for all processes on the first node.
* Variables usually are '''not propagated node-to-node.'''
; Consequences:
* You must [[HPC/Software/Module Setup | '''configure modules and $PATH''']] in your <code>$HOME/.bashrc</code> file to help mpirun and other mechanisms locate your programs on each compute node. Otherwise mpirun will fail as soon as more than one node is involved, either by not finding the binary itself or some of its required shared libraries, or more insidiously, using different shared libraries.
* Both the variable <code>$PBS_NODEFILE</code> and the file it points to are only available on the first node.
* To propagate (export) environment variables, use mpirun/mpiexec. However, there is no standard or quasi-standard across MPI flavors.
** [http://www.open-mpi.org/doc/v1.4/man1/mpirun.1.php#sect17 OpenMPI's mpirun] selectively exports some variables and supports the <code>-x</code> flag to export additional variables.
** Intel's mpiexec/mpiexec.hydra offer <code>-envlist ''NAME,NAME,...''</code> instead. See Sec. 2.4 in the ''Reference Manual'' at [http://software.intel.com/en-us/articles/intel-mpi-library-documentation/].

See also: [[HPC/Directories|directory layout]].

== Submitting jobs ==
=== Job Script ===
A job is described to Torque by a ''job script,'' which in most cases is a shell script containing directives for Torque.
* [[HPC/Submitting and Managing Jobs/Example Job Script | '''Example Job Script''']].

=== Submission ===
To submit a job, use Torque's [http://www.adaptivecomputing.com/resources/docs/torque/4-1-4/Content/topics/commands/qsub.htm qsub] command on a login node.
qsub [-A accountname] [options] ''jobfile''
To review qsub operation and options, consult the following:
man qsub
qsub --help

; Advanced usage:
* You ''can'' submit follow-up jobs from within a parent job script while it runs on a compute node, but it may be better to leverage [http://www.adaptivecomputing.com/resources/docs/torque/4-1-4/Content/topics/commands/qsub.htm#W job dependencies.]
* [http://www.adaptivecomputing.com/resources/docs/torque/4-1-4/Content/topics/2-jobs/requestingRes.htm Requesting Resources]
* Alternatively, use Moab's [http://www.adaptivecomputing.com/resources/docs/mwm/6-1-9/Content/commands/msub.html msub] command.

=== Queues ===
See [[HPC/Submitting and Managing Jobs/Queues and Policies | Queues and Policies]].

== Checking job status ==
Query your jobs using commands similar to those on other HPC clusters.
You can use either [http://www.adaptivecomputing.com/resources/docs/torque/4-1-4/Content/topics/12-appendices/commandsOverview.htm Torque commands]
or [http://www.adaptivecomputing.com/resources/docs/mwm/6-1-9/Content/a.gcommandoverview.html Moab commands].
Both access the same information, but through different routes, and offer different display styles.

* To get the regular status output:
[http://www.adaptivecomputing.com/resources/docs/torque/4-1-4/Content/topics/commands/qstat.htm qstat] [-u $USER]
[http://www.adaptivecomputing.com/resources/docs/mwm/6-1-9/Content/commands/showq.html showq] [-u $USER]
* To get an ''alternate'' format (e.g. showing job names):
qstat -a [-u $USER]
showq -n [-u $USER]
: Option <code>-u $USER</code> shows only your jobs. You can literally type <code>$USER</code> instead of your user name.
* To get ''full'' job status information such as submit arguments and run directories, and do not wrap long lines ("-1" is the digit "one"):
qstat -f [''jobnum''] [-1]

== Inspecting stuck jobs ==
To diagnose problems with "stuck" jobs, use <code>qstat -s</code>, or simply Moab's [http://www.adaptivecomputing.com/resources/docs/mwm/6-1-9/Content/commands/checkjob.html checkjob] command. The last few lines of output give ''past events'' of the job and its current status.
qstat -s ''jobnum''
checkjob -v ''jobnum''

To get estimates on when your job might start:
showstart [-e all] ''jobnum''

== Inspecting running jobs ==

=== Peeking at job output files ===
Each job has associated stdin and stderr streams which are gathered on its first node and are copied back only at job end time, according to the <code>#PBS</code> directives -e, -o, or -j [eo|oe]. If the user does explicitly capture any output in the script (using shell redirections <code>">"</code>) it can be difficult to assess a job's progress.

To inspect these streams ''while the job is running'', and also to review a job's script, use the <code>qpeek</code> command:
qpeek ''jobnum''
qpeek -eo ''jobnum''
qpeek -s ''jobnum''
Read its manual page for full documentation:
qpeek --man
This command is a Carbon re-implementation of a Torque "contrib" script.

To inspect a job's processes running on its compute nodes:
qpeek -p ''jobnum''

=== Which nodes does a running job use? ===
qstat -n [-1] ''jobnum''

== Changing job resources after submission ==

Use <code>qalter</code> to modify attributes of a job.
Only attributes specified on the command line will be changed.
If any of the specified attributes cannot be modified for any reason, ''none'' of that job's attributes will be modified.
qalter [-l ''resource''] […] ''jobnumber'' …
qalter -A ''accountspec'' ''jobnumber'' …
* Most attributes can be changed only ''before'' the job starts. Others, notably
 mail options and  job dependencies can be meaningfully adjusted after start.

; Advanced usage:
* ''Extending'' the <code>walltime</code> resource of an already running job requires Torque administrator permission. You yourself can extend prior to running, and shorten always. '''Warning:''' In either case, the new <code>walltime</code> limit will be measured from start, not as remaining time. If a new walltime limit is shorter than the elapsed time, the job will be killed right away.
* You cannot modify the job script itself after submission. You will have to remove and resubmit the job.
Contact me ([[User:Stern|stern]]) to arrange for interventions.

== Canceling (removing) jobs ==
To retract a queued job or kill an already running job:
qdel ''jobnumber''

Torque epilogue scripts will immediately remove remaining user processes and user files from the used compute nodes upon job termination, be it natural or after qdel.

== Declaring that a job requires licenses ==
A job that is to run software that requires a commercial license must declare such to the job scheduler (software that decides when and on which nodes the job could start).
The job scheduler monitors known licenses for availability, and so will know which and how many licenses are available at any given time.

'''Note:''' License availability checks do not happen instantaneously but rather by polling, roughly once every minute.
: This means that if you or another user happen to start a license-consuming application after the last check but before the relevant newly started job tries to access the same license, that job will terminate with a license failure.

=== One license type ===
To request a single license token of a single type, use a <code>gres</code> ("general resource") directive:
#PBS -l '''gres'''=''license_name''
To request several tokens of a type, place either a <code>:</code> or <code>+</code> as separator, followed by the number of tokens required:
#PBS -l '''gres'''=''license_name:count''

For compatibility, the resource type <code>software</code> is accepted, but ''not for multiple types'' of licenses.
#PBS -l '''software'''=''license_name''
#or
#PBS -l '''software'''=''license_name:count''

=== Several license types ===
To request several different licenses, use <code>gres</code> with list of license tokens and their optional counts separated by <code>'''%'''</code>:
#PBS -l gres=''license1_name'''''%'''''license2_name''
#PBS -l gres=''license1_name:count1'''''%'''''license2_name''

The <code>software</code> resource ''cannot'' request multiple license types.
(If several <code>software</code> resources are requested, only the ''last one'' would be active.)

== Interactive node access ==
There are two ways to interactively access nodes:
* Requesting an interactive job
qsub -I [-l nodes=…]
: This is a special Torque job submission that will wait for the requested nodes to become available, and then you will be dropped into a login shell on the first core of the first node. $PBS_NODEFILE is set as in regular (non-interactive) jobs.
* Side-access a node that already runs a job, such as to debug or inspect running processes or node-local files.
: You can use <code>ssh</code> to interactively access any compute node on which a job of yours is running (use <code>qsub -n ''jobnum''</code> to find out the node names). As soon as a node no longer runs at least ''one'' of your jobs, your ssh session to that node will be terminated. You may find the following commands helpful:
/usr/sbin/lsof -u $USER
/usr/sbin/lsof -c ''command_name''
/usr/sbin/lsof -p ''pid''
: - '''l'''i'''s'''t '''o'''pen '''f'''iles; [http://linux.die.net/man/8/lsof man page.]
ps -fu $USER
: - list your own processes.
psuser
: - a Carbon-specific wrapper for [http://linux.die.net/man/1/ps ps] to list user-only processes (yours and other regular users', excluding system processes).

== Job notification mails ==
You can request a notification message to be sent to you when a job begins, ends, or aborts. The message is rather terse and looks like this:
'''Subject:''' PBS Job 281399.sched1.carboncluster

PBS Job Id: 281399.sched1.carboncluster
Job Name: test.job
Exec host: n103/0
Execution terminated
Exit_status=0
resources_used.cput=00:03:00
resources_used.mem=123kb
resources_used.vmem=240kb
resources_used.walltime=00:03:04

In the above, Exit_status=0 reflects the shell exit status, where 0 means success.

The following means your job was terminated because it ran beyond its allocated walltime:
Exit_status=271
: The shell exit status is this number mod 256, in this case <code>271 % 256 = 15</code>, corresonding to <code>SIGTERM</code>.

== Recording where a job ran ==
The [[#Job notification mails]] explained above do not include $PBS_O_WORKDIR. You will have to arrange for yourself to record that information.
One way to do that would be to simply record all job numbers and their working directories at the beginning of each job:
<source lang="bash">
#!/bin/bash
#PBS -l nodes=…
#PBS -l walltime=…
#PBS -m bea

cd $PBS_O_WORKDIR
echo $PBS_JOBID $PWD >> $HOME/jobdirs

…
</source>
To query a job's directory, run:
grep ''jobnumber'' ~/jobdirs

== Sending mail while a job is running ==
It may be useful to have a job automatically "phone home" sometime after it started.
This can be accomplished at job submission time by arranging a mail to be sent, using a few added commands in the job script.

<syntaxhighlight lang="bash">
#!/bin/bash
#PBS …

cd $PBS_O_WORKDIR

# arrange notification mail
(
sleep 600 # time offset parameter, in seconds
tail -20 somefile.out | mailx -s "$PBS_JOBID: partial log" $USER
) &

# main work here
…
</syntaxhighlight>

Explanation:
* The commands inside <code>( … ) &</code> run in a background track aside the main work.
* The first command, <code>sleep</code>, delays further action (in this track only) by the specified time in seconds.
** To emphasize minutes, write <code>sleep $(( 10 * 60 ))</code> .
** To set the time relative to the walltime limit, leverage the <code>$PBS_WALLTIME</code> variable, e.g. as in <code>sleep $(( PBS_WALLTIME - ( ''minutes'' * 60 ) )).</code>
* The <code>tail</code> command is a stand-in for any inspection command you might want to run. An even simpler command could be <code>ls -lt</code>. You could also use several commands enclosed by parentheses, <code>( cmd1; cmd2; ... )</code>, to construct a more elaborate mail body.
* The <code>mailx</code> command, following the pipe symbol <code>|</code>, reads the output of your inspection command and sends you a mail with the subject specified by the <code>-s</code> option.
* If the main part of the job finishes before the pre-mail sleep is over no mail will be sent.

Note: The Moab scheduler also supports [http://www.adaptivecomputing.com/resources/docs/mwm/6-1-9/Content/topics/triggers/triggersIndex.html triggers], but they run in a different environment, not on an allocated node.

[[Category:HPC|Getting Started]]

HPC/Submitting and Managing Jobs

2026-02-17T20:40:19Z

Stern: /* Submitting and managing jobs */

[[Image:HPC job flow.png|right|400px|thumb|Processing flow of a typical job.]]
__TOC__
== Introduction ==
To accept and run jobs, Carbon uses the [http://en.wikipedia.org/wiki/TORQUE_Resource_Manager Torque Resource Manager],
which is a variant of [http://en.wikipedia.org/wiki/Portable_Batch_System PBS].
A separate software component called a [http://en.wikipedia.org/wiki/Job_scheduler ''scheduler''] chooses the nodes and the start time for jobs according to the job parameters, resource availability, and configuration. Carbon uses the [http://www.adaptivecomputing.com/resources/docs/mwm/6-1-9/help.htm Moab Workload Manager] as scheduler. The resources used by each job (primarily the CPU time) are tracked and managed by the
[http://www.adaptivecomputing.com/resources/docs/gold/index.php Gold Allocation Manager].

=== Job processing ===
A job consists of a [[#Job Script|job script]] and applications called therein. The figure at the top shows schematically how a typical job is processed.
Unless otherwise told, Torque will execute the job script on your behalf on the first node allocated by Moab,
in your <code>$HOME</code> directory, and typically using your login shell. It is the ''script's responsibility'' to:
* Change to a suitable working directory.
* Recognize and use the other nodes of the job.
** This is generally done by directing an [http://en.wikipedia.org/wiki/Message_Passing_Interface MPI] program to start on the nodes listed in the job's <code>$PBS_NODEFILE</code>.
** A job is perfectly free to run more than one program, such as several MPI programs in sequence.
* Set or propagate environment variables as needed to other nodes.

=== Environment variables ===
The handling of environment variables across the cores and nodes that a job uses varies with the capabilities and configuration choices of the various MPI implementations.


* Variables usually are '''propagated core-to-core''' for all processes on the first node.
* Variables usually are '''not propagated node-to-node.'''
; Consequences:
* You must [[HPC/Software/Module Setup | '''configure modules and $PATH''']] in your <code>$HOME/.bashrc</code> file to help mpirun and other mechanisms locate your programs on each compute node. Otherwise mpirun will fail as soon as more than one node is involved, either by not finding the binary itself or some of its required shared libraries, or more insidiously, using different shared libraries.
* Both the variable <code>$PBS_NODEFILE</code> and the file it points to are only available on the first node.
* To propagate (export) environment variables, use mpirun/mpiexec. However, there is no standard or quasi-standard across MPI flavors.
** [http://www.open-mpi.org/doc/v1.4/man1/mpirun.1.php#sect17 OpenMPI's mpirun] selectively exports some variables and supports the <code>-x</code> flag to export additional variables.
** Intel's mpiexec/mpiexec.hydra offer <code>-envlist ''NAME,NAME,...''</code> instead. See Sec. 2.4 in the ''Reference Manual'' at [http://software.intel.com/en-us/articles/intel-mpi-library-documentation/].

See also: [[HPC/Directories|directory layout]].

== Submitting jobs ==
=== Job Script ===
A job is described to Torque by a ''job script,'' which in most cases is a shell script containing directives for Torque.
* [[HPC/Submitting and Managing Jobs/Example Job Script | '''Example Job Script''']].

=== Submission ===
To submit a job, use Torque's [http://www.adaptivecomputing.com/resources/docs/torque/4-1-4/Content/topics/commands/qsub.htm qsub] command on a login node.
qsub [-A accountname] [options] ''jobfile''
To review qsub operation and options, consult the following:
man qsub
qsub --help

; Advanced usage:
* You ''can'' submit follow-up jobs from within a parent job script while it runs on a compute node, but it may be better to leverage [http://www.adaptivecomputing.com/resources/docs/torque/4-1-4/Content/topics/commands/qsub.htm#W job dependencies.]
* [http://www.adaptivecomputing.com/resources/docs/torque/4-1-4/Content/topics/2-jobs/requestingRes.htm Requesting Resources]
* Alternatively, use Moab's [http://www.adaptivecomputing.com/resources/docs/mwm/6-1-9/Content/commands/msub.html msub] command.

=== Queues ===
See [[HPC/Submitting and Managing Jobs/Queues and Policies | Queues and Policies]].

== Checking job status ==
Query your jobs using commands similar to those on other HPC clusters.
You can use either [http://www.adaptivecomputing.com/resources/docs/torque/4-1-4/Content/topics/12-appendices/commandsOverview.htm Torque commands]
or [http://www.adaptivecomputing.com/resources/docs/mwm/6-1-9/Content/a.gcommandoverview.html Moab commands].
Both access the same information, but through different routes, and offer different display styles.

* To get the regular status output:
[http://www.adaptivecomputing.com/resources/docs/torque/4-1-4/Content/topics/commands/qstat.htm qstat] [-u $USER]
[http://www.adaptivecomputing.com/resources/docs/mwm/6-1-9/Content/commands/showq.html showq] [-u $USER]
* To get an ''alternate'' format (e.g. showing job names):
qstat -a [-u $USER]
showq -n [-u $USER]
: Option <code>-u $USER</code> shows only your jobs. You can literally type <code>$USER</code> instead of your user name.
* To get ''full'' job status information such as submit arguments and run directories, and do not wrap long lines ("-1" is the digit "one"):
qstat -f [''jobnum''] [-1]

== Inspecting "stuck" jobs ==
To diagnose problems with "stuck" jobs, use <code>qstat -s</code>, or simply Moab's [http://www.adaptivecomputing.com/resources/docs/mwm/6-1-9/Content/commands/checkjob.html checkjob] command. The last few lines of output give ''past events'' of the job and its current status.
qstat -s ''jobnum''
checkjob -v ''jobnum''

To get estimates on when your job might start:
showstart [-e all] ''jobnum''

== Inspecting running jobs ==

=== Peeking at job output files ===
Each job has associated stdin and stderr streams which are gathered on its first node and are copied back only at job end time, according to the <code>#PBS</code> directives -e, -o, or -j [eo|oe]. If the user does explicitly capture any output in the script (using shell redirections <code>">"</code>) it can be difficult to assess a job's progress.

To inspect these streams ''while the job is running'', and also to review a job's script, use the <code>qpeek</code> command:
qpeek ''jobnum''
qpeek -eo ''jobnum''
qpeek -s ''jobnum''
Read its manual page for full documentation:
qpeek --man
This command is a Carbon re-implementation of a Torque "contrib" script.

To inspect a job's processes running on its compute nodes:
qpeek -p ''jobnum''

=== Which nodes does a running job use? ===
qstat -n [-1] ''jobnum''

== Changing job resources after submission ==

Use <code>qalter</code> to modify attributes of a job.
Only attributes specified on the command line will be changed.
If any of the specified attributes cannot be modified for any reason, ''none'' of that job's attributes will be modified.
qalter [-l ''resource''] […] ''jobnumber'' …
qalter -A ''accountspec'' ''jobnumber'' …
* Most attributes can be changed only ''before'' the job starts. Others, notably
 mail options and  job dependencies can be meaningfully adjusted after start.

; Advanced usage:
* ''Extending'' the <code>walltime</code> resource of an already running job requires Torque administrator permission. You yourself can extend prior to running, and shorten always. '''Warning:''' In either case, the new <code>walltime</code> limit will be measured from start, not as remaining time. If a new walltime limit is shorter than the elapsed time, the job will be killed right away.
* You cannot modify the job script itself after submission. You will have to remove and resubmit the job.
Contact me ([[User:Stern|stern]]) to arrange for interventions.

== Canceling (removing) jobs ==
To retract a queued job or kill an already running job:
qdel ''jobnumber''

Torque epilogue scripts will immediately remove remaining user processes and user files from the used compute nodes upon job termination, be it natural or after qdel.

== Declaring that a job requires licenses ==
A job that is to run software that requires a commercial license must declare such to the job scheduler (software that decides when and on which nodes the job could start).
The job scheduler monitors known licenses for availability, and so will know which and how many licenses are available at any given time.

'''Note:''' License availability checks do not happen instantaneously but rather by polling, roughly once every minute.
: This means that if you or another user happen to start a license-consuming application after the last check but before the relevant newly started job tries to access the same license, that job will terminate with a license failure.

=== One license type ===
To request a single license token of a single type, use a <code>gres</code> ("general resource") directive:
#PBS -l '''gres'''=''license_name''
To request several tokens of a type, place either a <code>:</code> or <code>+</code> as separator, followed by the number of tokens required:
#PBS -l '''gres'''=''license_name:count''

For compatibility, the resource type <code>software</code> is accepted, but ''not for multiple types'' of licenses.
#PBS -l '''software'''=''license_name''
#or
#PBS -l '''software'''=''license_name:count''

=== Several license types ===
To request several different licenses, use <code>gres</code> with list of license tokens and their optional counts separated by <code>'''%'''</code>:
#PBS -l gres=''license1_name'''''%'''''license2_name''
#PBS -l gres=''license1_name:count1'''''%'''''license2_name''

The <code>software</code> resource ''cannot'' request multiple license types.
(If several <code>software</code> resources are requested, only the ''last one'' would be active.)

== Interactive node access ==
There are two ways to interactively access nodes:
* Requesting an interactive job
qsub -I [-l nodes=…]
: This is a special Torque job submission that will wait for the requested nodes to become available, and then you will be dropped into a login shell on the first core of the first node. $PBS_NODEFILE is set as in regular (non-interactive) jobs.
* Side-access a node that already runs a job, such as to debug or inspect running processes or node-local files.
: You can use <code>ssh</code> to interactively access any compute node on which a job of yours is running (use <code>qsub -n ''jobnum''</code> to find out the node names). As soon as a node no longer runs at least ''one'' of your jobs, your ssh session to that node will be terminated. You may find the following commands helpful:
/usr/sbin/lsof -u $USER
/usr/sbin/lsof -c ''command_name''
/usr/sbin/lsof -p ''pid''
: - '''l'''i'''s'''t '''o'''pen '''f'''iles; [http://linux.die.net/man/8/lsof man page.]
ps -fu $USER
: - list your own processes.
psuser
: - a Carbon-specific wrapper for [http://linux.die.net/man/1/ps ps] to list user-only processes (yours and other regular users', excluding system processes).

== Job notification mails ==
You can request a notification message to be sent to you when a job begins, ends, or aborts. The message is rather terse and looks like this:
'''Subject:''' PBS Job 281399.sched1.carboncluster

PBS Job Id: 281399.sched1.carboncluster
Job Name: test.job
Exec host: n103/0
Execution terminated
Exit_status=0
resources_used.cput=00:03:00
resources_used.mem=123kb
resources_used.vmem=240kb
resources_used.walltime=00:03:04

In the above, Exit_status=0 reflects the shell exit status, where 0 means success.

The following means your job was terminated because it ran beyond its allocated walltime:
Exit_status=271
: The shell exit status is this number mod 256, in this case <code>271 % 256 = 15</code>, corresonding to <code>SIGTERM</code>.

== Recording where a job ran ==
The [[#Job notification mails]] explained above do not include $PBS_O_WORKDIR. You will have to arrange for yourself to record that information.
One way to do that would be to simply record all job numbers and their working directories at the beginning of each job:
<source lang="bash">
#!/bin/bash
#PBS -l nodes=…
#PBS -l walltime=…
#PBS -m bea

cd $PBS_O_WORKDIR
echo $PBS_JOBID $PWD >> $HOME/jobdirs

…
</source>
To query a job's directory, run:
grep ''jobnumber'' ~/jobdirs

== Sending mail while a job is running ==
It may be useful to have a job automatically "phone home" sometime after it started.
This can be accomplished at job submission time by arranging a mail to be sent, using a few added commands in the job script.

<syntaxhighlight lang="bash">
#!/bin/bash
#PBS …

cd $PBS_O_WORKDIR

# arrange notification mail
(
sleep 600 # time offset parameter, in seconds
tail -20 somefile.out | mailx -s "$PBS_JOBID: partial log" $USER
) &

# main work here
…
</syntaxhighlight>

Explanation:
* The commands inside <code>( … ) &</code> run in a background track aside the main work.
* The first command, <code>sleep</code>, delays further action (in this track only) by the specified time in seconds.
** To emphasize minutes, write <code>sleep $(( 10 * 60 ))</code> .
** To set the time relative to the walltime limit, leverage the <code>$PBS_WALLTIME</code> variable, e.g. as in <code>sleep $(( PBS_WALLTIME - ( ''minutes'' * 60 ) )).</code>
* The <code>tail</code> command is a stand-in for any inspection command you might want to run. An even simpler command could be <code>ls -lt</code>. You could also use several commands enclosed by parentheses, <code>( cmd1; cmd2; ... )</code>, to construct a more elaborate mail body.
* The <code>mailx</code> command, following the pipe symbol <code>|</code>, reads the output of your inspection command and sends you a mail with the subject specified by the <code>-s</code> option.
* If the main part of the job finishes before the pre-mail sleep is over no mail will be sent.

Note: The Moab scheduler also supports [http://www.adaptivecomputing.com/resources/docs/mwm/6-1-9/Content/topics/triggers/triggersIndex.html triggers], but they run in a different environment, not on an allocated node.

[[Category:HPC|Getting Started]]

HPC/Submitting and Managing Jobs

2026-02-17T20:35:44Z

Stern: /* Submitting and managing jobs */

[[Image:HPC job flow.png|right|400px|thumb|Processing flow of a typical job.]]
__TOC__
== Introduction ==
To accept and run jobs, Carbon uses the [http://en.wikipedia.org/wiki/TORQUE_Resource_Manager Torque Resource Manager],
which is a variant of [http://en.wikipedia.org/wiki/Portable_Batch_System PBS].
A separate software component called a [http://en.wikipedia.org/wiki/Job_scheduler ''scheduler''] chooses the nodes and the start time for jobs according to the job parameters, resource availability, and configuration. Carbon uses the [http://www.adaptivecomputing.com/resources/docs/mwm/6-1-9/help.htm Moab Workload Manager] as scheduler. The resources used by each job (primarily the CPU time) are tracked and managed by the
[http://www.adaptivecomputing.com/resources/docs/gold/index.php Gold Allocation Manager].

=== Job processing ===
A job consists of a [[#Job Script|job script]] and applications called therein. The figure at the top shows schematically how a typical job is processed.
Unless otherwise told, Torque will execute the job script on your behalf on the first node allocated by Moab,
in your <code>$HOME</code> directory, and typically using your login shell. It is the ''script's responsibility'' to:
* Change to a suitable working directory.
* Recognize and use the other nodes of the job.
** This is generally done by directing an [http://en.wikipedia.org/wiki/Message_Passing_Interface MPI] program to start on the nodes listed in the job's <code>$PBS_NODEFILE</code>.
** A job is perfectly free to run more than one program, such as several MPI programs in sequence.
* Set or propagate environment variables as needed to other nodes.

=== Environment variables ===
The handling of environment variables across the cores and nodes that a job uses varies with the capabilities and configuration choices of the various MPI implementations.


* Variables usually are '''propagated core-to-core''' for all processes on the first node.
* Variables usually are '''not propagated node-to-node.'''
; Consequences:
* You must [[HPC/Software/Module Setup | '''configure modules and $PATH''']] in your <code>$HOME/.bashrc</code> file to help mpirun and other mechanisms locate your programs on each compute node. Otherwise mpirun will fail as soon as more than one node is involved, either by not finding the binary itself or some of its required shared libraries, or more insidiously, using different shared libraries.
* Both the variable <code>$PBS_NODEFILE</code> and the file it points to are only available on the first node.
* To propagate (export) environment variables, use mpirun/mpiexec. However, there is no standard or quasi-standard across MPI flavors.
** [http://www.open-mpi.org/doc/v1.4/man1/mpirun.1.php#sect17 OpenMPI's mpirun] selectively exports some variables and supports the <code>-x</code> flag to export additional variables.
** Intel's mpiexec/mpiexec.hydra offer <code>-envlist ''NAME,NAME,...''</code> instead. See Sec. 2.4 in the ''Reference Manual'' at [http://software.intel.com/en-us/articles/intel-mpi-library-documentation/].

See also: [[HPC/Directories|directory layout]].

== Submitting and managing jobs ==
=== Job Script ===
A job is described to Torque by a ''job script,'' which in most cases is a shell script containing directives for Torque.
* [[HPC/Submitting and Managing Jobs/Example Job Script | '''Example Job Script''']].

=== Submission ===
To submit a job, use Torque's [http://www.adaptivecomputing.com/resources/docs/torque/4-1-4/Content/topics/commands/qsub.htm qsub] command on a login node.
qsub [-A accountname] [options] ''jobfile''
To review qsub operation and options, consult the following:
man qsub
qsub --help

; Advanced usage:
* You ''can'' submit follow-up jobs from within a parent job script while it runs on a compute node, but it may be better to leverage [http://www.adaptivecomputing.com/resources/docs/torque/4-1-4/Content/topics/commands/qsub.htm#W job dependencies.]
* [http://www.adaptivecomputing.com/resources/docs/torque/4-1-4/Content/topics/2-jobs/requestingRes.htm Requesting Resources]
* Alternatively, use Moab's [http://www.adaptivecomputing.com/resources/docs/mwm/6-1-9/Content/commands/msub.html msub] command.

=== Queues ===
See [[HPC/Submitting and Managing Jobs/Queues and Policies | Queues and Policies]].

=== Checking job status ===
Query your jobs using commands similar to those on other HPC clusters.
You can use either [http://www.adaptivecomputing.com/resources/docs/torque/4-1-4/Content/topics/12-appendices/commandsOverview.htm Torque commands]
or [http://www.adaptivecomputing.com/resources/docs/mwm/6-1-9/Content/a.gcommandoverview.html Moab commands].
Both access the same information, but through different routes, and offer different display styles.

* To get the regular status output:
[http://www.adaptivecomputing.com/resources/docs/torque/4-1-4/Content/topics/commands/qstat.htm qstat] [-u $USER]
[http://www.adaptivecomputing.com/resources/docs/mwm/6-1-9/Content/commands/showq.html showq] [-u $USER]
* To get an ''alternate'' format (e.g. showing job names):
qstat -a [-u $USER]
showq -n [-u $USER]
: Option <code>-u $USER</code> shows only your jobs. You can literally type <code>$USER</code> instead of your user name.
* To get ''full'' job status information such as submit arguments and run directories, and do not wrap long lines ("-1" is the digit "one"):
qstat -f [''jobnum''] [-1]

=== Inspect stuck jobs ===
To diagnose problems with "stuck" jobs, use <code>qstat -s</code>, or simply Moab's [http://www.adaptivecomputing.com/resources/docs/mwm/6-1-9/Content/commands/checkjob.html checkjob] command. The last few lines of output give ''past events'' of the job and its current status.
qstat -s ''jobnum''
checkjob -v ''jobnum''

To get estimates on when your job might start:
showstart [-e all] ''jobnum''

=== Inspect running jobs ===
==== Peeking at job output files ====
Each job has associated stdin and stderr streams which are gathered on its first node and are copied back only at job end time, according to the <code>#PBS</code> directives -e, -o, or -j [eo|oe]. If the user does explicitly capture any output in the script (using shell redirections <code>">"</code>) it can be difficult to assess a job's progress.

To inspect these streams ''while the job is running'', and also to review a job's script, use the <code>qpeek</code> command:
qpeek ''jobnum''
qpeek -eo ''jobnum''
qpeek -s ''jobnum''
Read its manual page for full documentation:
qpeek --man
This command is a Carbon re-implementation of a Torque "contrib" script.

To inspect a job's processes running on its compute nodes:
qpeek -p ''jobnum''

==== Which nodes does a running job use? ====
qstat -n [-1] ''jobnum''

=== Changing jobs after submission ===
Use <code>[http://www.adaptivecomputing.com/resources/docs/torque/4-1-4/Content/topics/commands/qalter.htm qalter]</code> to modify attributes of a job.
Only attributes specified on the command line will be changed.
If any of the specified attributes cannot be modified for any reason, ''none'' of that job's attributes will be modified.
qalter [-l ''resource''] […] ''jobnumber'' …
qalter -A ''accountspec'' ''jobnumber'' …
* Most attributes can be changed only ''before'' the job starts. Others, notably [http://www.adaptivecomputing.com/resources/docs/torque/4-1-4/Content/topics/commands/qsub.htm#-m mail options] and [http://www.adaptivecomputing.com/resources/docs/torque/4-1-4/Content/topics/commands/qsub.htm#-W job dependencies] can be meaningfully adjusted after start.

; Advanced usage:
* ''Extending'' the <code>walltime</code> resource of an already running job requires Torque administrator permission. You yourself can extend prior to running, and shorten always. '''Warning:''' In either case, the new <code>walltime</code> limit will be measured from start, not as remaining time. If a new walltime limit is shorter than the elapsed time, the job will be killed right away.
* You cannot modify the job script itself after submission. You will have to remove and resubmit the job.
Contact me ([[User:Stern|stern]]) to arrange for interventions.

=== Removing jobs ===
To retract a queued job or kill an already running job:
qdel ''jobnumber''
[http://www.adaptivecomputing.com/resources/docs/torque/4-1-4/Content/topics/12-appendices/prologueAndEpliogueScripts.htm Torque epilogue scripts] will immediately remove remaining user processes and user files from the used compute nodes upon job termination, be it natural or after qdel.

== Declaring that a job requires licenses ==
A job that is to run software that requires a commercial license must declare such to the job scheduler (software that decides when and on which nodes the job could start).
The job scheduler monitors known licenses for availability, and so will know which and how many licenses are available at any given time.

'''Note:''' License availability checks do not happen instantaneously but rather by polling, roughly once every minute.
: This means that if you or another user happen to start a license-consuming application after the last check but before the relevant newly started job tries to access the same license, that job will terminate with a license failure.

=== One license type ===
To request a single license token of a single type, use a <code>gres</code> ("general resource") directive:
#PBS -l '''gres'''=''license_name''
To request several tokens of a type, place either a <code>:</code> or <code>+</code> as separator, followed by the number of tokens required:
#PBS -l '''gres'''=''license_name:count''

For compatibility, the resource type <code>software</code> is accepted, but ''not for multiple types'' of licenses.
#PBS -l '''software'''=''license_name''
#or
#PBS -l '''software'''=''license_name:count''

=== Several license types ===
To request several different licenses, use <code>gres</code> with list of license tokens and their optional counts separated by <code>'''%'''</code>:
#PBS -l gres=''license1_name'''''%'''''license2_name''
#PBS -l gres=''license1_name:count1'''''%'''''license2_name''

The <code>software</code> resource ''cannot'' request multiple license types.
(If several <code>software</code> resources are requested, only the ''last one'' would be active.)

== Interactive node access ==
There are two ways to interactively access nodes:
* Requesting an interactive job
qsub -I [-l nodes=…]
: This is a special Torque job submission that will wait for the requested nodes to become available, and then you will be dropped into a login shell on the first core of the first node. $PBS_NODEFILE is set as in regular (non-interactive) jobs.
* Side-access a node that already runs a job, such as to debug or inspect running processes or node-local files.
: You can use <code>ssh</code> to interactively access any compute node on which a job of yours is running (use <code>qsub -n ''jobnum''</code> to find out the node names). As soon as a node no longer runs at least ''one'' of your jobs, your ssh session to that node will be terminated. You may find the following commands helpful:
/usr/sbin/lsof -u $USER
/usr/sbin/lsof -c ''command_name''
/usr/sbin/lsof -p ''pid''
: - '''l'''i'''s'''t '''o'''pen '''f'''iles; [http://linux.die.net/man/8/lsof man page.]
ps -fu $USER
: - list your own processes.
psuser
: - a Carbon-specific wrapper for [http://linux.die.net/man/1/ps ps] to list user-only processes (yours and other regular users', excluding system processes).

== Job notification mails ==
You can request a notification message to be sent to you when a job begins, ends, or aborts. The message is rather terse and looks like this:
'''Subject:''' PBS Job 281399.sched1.carboncluster

PBS Job Id: 281399.sched1.carboncluster
Job Name: test.job
Exec host: n103/0
Execution terminated
Exit_status=0
resources_used.cput=00:03:00
resources_used.mem=123kb
resources_used.vmem=240kb
resources_used.walltime=00:03:04

In the above, Exit_status=0 reflects the shell exit status, where 0 means success.

The following means your job was terminated because it ran beyond its allocated walltime:
Exit_status=271
: The shell exit status is this number mod 256, in this case <code>271 % 256 = 15</code>, corresonding to <code>SIGTERM</code>.

== Recording where a job ran ==
The [[#Job notification mails]] explained above do not include $PBS_O_WORKDIR. You will have to arrange for yourself to record that information.
One way to do that would be to simply record all job numbers and their working directories at the beginning of each job:
<source lang="bash">
#!/bin/bash
#PBS -l nodes=…
#PBS -l walltime=…
#PBS -m bea

cd $PBS_O_WORKDIR
echo $PBS_JOBID $PWD >> $HOME/jobdirs

…
</source>
To query a job's directory, run:
grep ''jobnumber'' ~/jobdirs

== Sending mail while a job is running ==
It may be useful to have a job automatically "phone home" sometime after it started.
This can be accomplished at job submission time by arranging a mail to be sent, using a few added commands in the job script.

<syntaxhighlight lang="bash">
#!/bin/bash
#PBS …

cd $PBS_O_WORKDIR

# arrange notification mail
(
sleep 600 # time offset parameter, in seconds
tail -20 somefile.out | mailx -s "$PBS_JOBID: partial log" $USER
) &

# main work here
…
</syntaxhighlight>

Explanation:
* The commands inside <code>( … ) &</code> run in a background track aside the main work.
* The first command, <code>sleep</code>, delays further action (in this track only) by the specified time in seconds.
** To emphasize minutes, write <code>sleep $(( 10 * 60 ))</code> .
** To set the time relative to the walltime limit, leverage the <code>$PBS_WALLTIME</code> variable, e.g. as in <code>sleep $(( PBS_WALLTIME - ( ''minutes'' * 60 ) )).</code>
* The <code>tail</code> command is a stand-in for any inspection command you might want to run. An even simpler command could be <code>ls -lt</code>. You could also use several commands enclosed by parentheses, <code>( cmd1; cmd2; ... )</code>, to construct a more elaborate mail body.
* The <code>mailx</code> command, following the pipe symbol <code>|</code>, reads the output of your inspection command and sends you a mail with the subject specified by the <code>-s</code> option.
* If the main part of the job finishes before the pre-mail sleep is over no mail will be sent.

Note: The Moab scheduler also supports [http://www.adaptivecomputing.com/resources/docs/mwm/6-1-9/Content/topics/triggers/triggersIndex.html triggers], but they run in a different environment, not on an allocated node.

[[Category:HPC|Getting Started]]

HPC/Applications/comsol

2026-02-04T23:30:39Z

Stern: /* The modes to run COMSOL */

== Introduction ==
The [http://www.comsol.com/products/multiphysics/ '''COMSOL Multiphysics'''] application suite
is available for Mac, Windows, and Linux.

== Access ==
[[Image:HPC Comsol licensed modules.png|right|450px|thumb|COMSOL features licensed at CNM as of 2024-10.]]
=== COMSOL Multiphysics ===
Access to '''COMSOL Multiphysics''', the main application with both a graphical user interface and batch computing capabilities, will be granted '''''upon request''''' .

You must be:
* An '''''Argonne employee'''''  in one of the following roles:
** '''''NST staff member'''''  (including PostDocs, joint appointments, and students at NST).
** '''''CNM Facility User'''''  (including ''non-NST Argonne employees''), with an active proposal that has COMSOL in scope. Please see [[HPC/FAQ#Access_and_proposals]].

The set of available COMSOL modules that we have licensed is shown at right.

* The native desktop version can be installed only on ''Argonne-owned'' computers (laptop or desktop). Please contact [mailto:help@anl.gov?Subject=COMSOL%20license%20access%20request CNM support]. Make sure to review [http://www.comsol.com/support/updates/ Product Updates].
* To run COMSOL on our HPC system ''Carbon'', as GUI or as batch computing jobs, your proposal must have ''Carbon'' access in scope.

=== COMSOL Server ===
''CNM Facility Users'' who are not Argonne employees are not eligible to access
the full '''COMSOL Multiphysics''' product, in compliance with its licensing terms and conditions.
However, the [https://www.comsol.com/comsol-server '''COMSOL Server'''] product offers an alternative
albeit more narrow means of access.

All users, including '''''non-Argonne users''''' may be able to run, over a web interface in their browser,
pre-parameterized "fixed-model" simulations that were made with the '''COMSOL Application Builder''' (requires Windows).

To inquire about using COMSOL server applications,
please contact us.

We will review possible purchase paths for COMSOL Server, and the ''considerable effort'' likely required by users and the CNM to develop and deploy a COMSOL application.

== Support ==
* For questions on how to run or start COMSOL:
*: Submit an [[HPC/Support | '''IT support request''']]
* For questions related to most COMSOL data or scripts, contact COMSOL support at:
*: [mailto:support@comsol.com '''support'''@comsol.com]
* '''Exception:''' If your support question might contain data that is subject to '''US Export Control''' regulations, contact instead:
*: [mailto:us-export-control@us.comsol.com '''us-export-control'''@'''us'''.comsol.com]

== The modes to run COMSOL ==
There are several ways of running COMSOL.

=== Running the desktop application ===
To edit models and analyze results, use the COMSOL graphical user interface (GUI). You can run it in several ways, each explained in more detail in the following sections.

# Run on Carbon, display on your desktop using the [http://en.wikipedia.org/wiki/X_Window_System X Window system (X11)].
#* Requires an ''X11 Server'' application running on your client machine, and SSH X11 forwarding.
#* The X11 protocol can be slow over LAN.
# Run on Carbon within a [[HPC/Network Access/Virtual Desktop | virtual desktop environment (VNC)]].
#* Requires a VNC client on your machine.
#* It can be confusing where VNC and the app runs.
#* The virtual desktop presents a (deliberatly) sparse windowing environment – Copy&Paste can be difficult to use.
# Run on your Argonne-provided computer, checking out Carbon's license remotely.
#* Requires a full installation of COMSOL on your desktoo or laptop computer.
#* You will require access to the COMSOL license server running on Carbon, either through VPN (fairly easy) or SSH tunneling (can be intimidating for first-time users). See below for details.

=== Running a batch calculation ===
When the computational stage of models takes longer than a few minutes to complete, it will be beneficial to submit save the model file and run it in parallel on Carbon.
See the [[#Batch jobs|section below]].

== Use with X11 ==
For casual use, and with [[HPC/Network Access/SSH Tunnel Setup on Linux and MacOS | X11 tunneling to Carbon]] set up, simply add to your <code>~/.bashrc</code> file on Carbon:
module load comsol

Then, from the command line on a Carbon login node, start the COMSOL GUI:
comsol &

License configuration is automatic on Carbon.

Comsol will use a number of cores to run calculations multithreaded by default. (On Carbon, COMSOL_NUM_THREADS is set.)

'''Quit the application''' as soon as you are done. Only one user can use Comsol at a time.

== Use with VNC ==
See general [[HPC/Network Access/Virtual Desktop | instructions on using VNC]], and the COMSOL example therein.

== Native desktop version ==

To get the fastest response from the COMSOL GUI, run a native COMSOL version on your desktop machine or laptop, called ''client machine'' below.

Enter license details either in the COMSOL Setup configuration dialog.

== Native desktop version — License configuration ==
A client's COMSOL license needs to be configured depending on how the client reaches CNM network resources. Follow the appropriate section:

* [[#License configuration when inside CNM or on CNM VPN|License configuration when '''inside CNM''' or on '''CNM VPN''']]
: Requires that the user's machine is logically within the network of CNM/NST, either by a wired/auth WiFi connection in the building or by VPN under a CNM/NST profile.
* [[#License configuration when outside CNM – requires SSH Tunneling|License configuration when '''outside CNM''' – requires SSH Tunneling]]
: ''Non-NST staff'' usually have a VPN profile that places the machine logically [[#License configuration when outside CNM – requires SSH Tunneling|outside of NST's network.]]

=== License configuration when inside CNM or on CNM VPN ===
==== Ensure correct DNS search domains ====
* Inspect the client machine's network configuration.
* Ensure that '''two''' entries are listed as DNS Search Domains for the current network profile:
nst.anl.gov
cnm.anl.gov
: (The order of entries does not matter for the purposes considered here.)
* If the client machine is ''wired to CNM networks'', the domains are normally provided correctly by DHCP.
* If the client machine is ''connected over VPN'', you may need edit the respective network profile.
:* On a Mac, find the VPN configuration details under System Preferences, Network, Advanced… . You may need to unlock the Network panel first before you can make changes under "Advanced…". It appears that those changes are being saved and will be activated automatically the next time that this VPN session gets started.
: [[Image:VNC_add_cnm_to_DNS_1.png|400px]] [[Image:VNC_add_cnm_to_DNS_2.png|400px]]
:* On Windows or Linux machines, take ''analogous'' steps, using the platform's own configuration tools.

* Exclude license manager components from client desktop installation. (Those are all handled server-side.)

==== License format ====
* Floating Network License (FNL), "Three-server redundancy"

==== License servers ====
* On client machines, while running the COMSOL installer or license configuration, specify license servers and ports as follows:
1718 clicense1
1718 clicense2
1718 clicense3
: '''Note:''' Use ''short host names only''. (Technical Reason: The license must be usable both on Carbon's internal network as well as on divisional networks).
1718 clicense1<strike>.cnm.anl.gov</strike>
1718 clicense2<strike>.cnm.anl.gov</strike>
1718 clicense3<strike>.cnm.anl.gov</strike>

'''Advanced:''' A client machine's COMSOL license file <code><COMSOLDIR>/Multiphysics/license/license.dat</code> should look as follows:
<source lang="make">
SERVER clicense1 ANY 1718
SERVER clicense2 ANY 1718
SERVER clicense3 ANY 1718
USE_SERVER
</source>

<code><COMSOLDIR></code> is located by default at:
* '''Windows''': C:\Program files\COMSOL\COMSOL''nn''
* '''Linux''': /usr/local/comsol''nn''
* '''Mac''': /Applications/COMSOL''nn''
("''nn''" is the application's abbreviated version number.)

==== Network Troubleshooting ====
When using the license access model descibed here (as opposed to tunneled over SSH), ensure the following:
* Verify that short host names can be resolved to IP addresses. Either one of the following commands should produce sensible output:
host clicense1
: or:
ping -c 3 clicense1
* Verify that ''both license ports on the same server'' can be reached from your client machine:
telnet '''clicense1''' 1718
telnet '''clicense1''' 1719
: or:
telnet '''clicense2''' 1718
telnet '''clicense2''' 1719
: A ''successful'' connection looks like:
Trying 146.139.x.y...
Connected to clicense1.cnm.anl.gov.
Escape character is '^]'.
:: Do not enter any data. Instead, close each connection by hitting <code>Ctrl</code><code>]</code> (closing square bracket), followed by <code>Ctrl</code><code>D</code>.
: A ''failed'' connection gives one of these errors:
telnet: connect to address 146.139.x.y: Operation timed out
: or:
telnet: connect to address 146.139.x.y: Connection refused
:: In these cases, ask that your IP firewall configuration be reviewed and changed.

=== License configuration when outside CNM – requires SSH Tunneling ===
If you're comfortable with ssh and tunneling, set up client licensing as follows:

==== Configure COMSOL ====
The following needs to be done just once, typically during COMSOL installation:

In the COMSOL Installer, or in COMSOL Preferences, enter:
* '''License format:''' Floating Network License (FNL), "Port number and Hostname"
:: "Three-server redundancy" cannot be used because is cannot (easily) be accessed over SSH. Configure with just one server, and in rare tweak SSH to tunnel to the other server.
* '''Server:''' <code>127.0.0.1</code>
:: Do not use "localhost", as COMSOL interprets this incorrectly to mean that your machine ''is'' a license server.

* '''Port:''' <code>1718</code>


==== Provide server connection ====
This step needs to be done prior to running COMSOL in your desktop session:
* Connect to <code>mega</code> with the following ssh command:
ssh -L 1718:'''clicense1''':1718 -L 1719:'''clicense1''':1719 mega.cnm.anl.gov
* Leave the command running between invocations of COMSOL.

==== Run COMSOL ====
* Ensure your connection to "mega" is still running. If not, restart it.
* Start the COMSOL application.

** If you get an error that the number of licenses is "exhausted", that is confirmation that you correctly reached the license server, though other users are using licenses.
** If the license access attempt takes a long time or hangs, the service might temporarily run on a secondary server. In this case, stop the mega session and restart it, connecting to the alternate license server:
ssh -L 1718:clicense'''2''':1718 -L 1719:clicense'''2''':1719 mega.cnm.anl.gov

== License Troubleshooting ==
The following are typical error messages.
=== Could not obtain license for COMSOL Multiphysics GUI ===
* Error details:
License error: -4.
'''Licensed number of users already reached.'''
Feature: COMSOLGUI
License path: /Users/stern/Applications/COMSOL52a/license/license.dat:
FlexNet Licensing error:-4,132
* Reason: The connection to the license server succeeded, but the license token(s) you requested are in use.
* Solution: Wait a bit, or [http://carbonmgmt/ganglia/lic.php identify who's using COMSOL] and politely ask the user when they might be finished with their work.

=== License server machine is down or not responding ===
* Error details:
License error: -96.
See the system administrator about starting the license server system, or
make sure you're referring to the right host (see LM_LICENSE_FILE).
Feature: SERIAL
Hostname: clicense3
License path: /Users/stern/Applications/COMSOL52a/license/license.dat:
FlexNet Licensing error:-96,7
* Reason: The license servers cannot be reached at all. The <code>ping</code> or <code>host</code> tests mentioned above fail.
* Solution: Review the client license file and DNS lookup domains.

=== License file does not support this version ===
* Error details:
Feature: SERIAL
'''Application version > License version: 5.21 > 3.5'''
License path: /Users/stern/Applications/COMSOL52a/license/license.dat:
FlexNet Licensing error:-21,126
* Reason: Only the first of two steps for connecting to the license servers succeeds, which may happen when the redundant license service runs on the alternate server instead of the one you're connected to.
* Solution: When using SSH, try <code>clicense2</code> instead of <code>clicense1</code>, or vice versa. Revisit your <code>~/.ssh/config</code> file or use the alternate command line as shown in the SSH section above.

== Multiple CPU cores ==
Ways to start multi-core (from http://www.comsol.com/support/knowledgebase/1096/)
comsol -np 8

export COMSOL_NUM_THREADS=8 # automatically provided on ''Carbon''
comsol
* COMSOL does not benefit from hyperthreading (oversubscribing the physical core count).

== Batch jobs ==

* http://www.comsol.com/support/knowledgebase/1001/ .
* [[Media:HPC 2012-05-16 Comsol parallel usage.pdf]] (Modified from http://www.comsol.com/support/knowledgebase/1001/files/4027/cluster_install_linux_42a.pptx – Appendix slide, PBS)
* '''Note:''' MPD is not supported on Carbon (I patched the main comsol script), and no longer necessary at all in Comsol-4.3.

=== Sample Job file ===
See:
$COMSOL_HOME/sample.job

== Important COMSOL Support Articles ==
* Toplevel suport page: https://www.comsol.com/support

=== KB Articles ===
* [https://www.comsol.com/support/knowledgebase/1001 Running COMSOL® in parallel on clusters]
* [https://www.comsol.com/support/knowledgebase/1250 Running parametric sweeps, batch sweeps, and cluster sweeps from the command line]
* [https://www.comsol.com/support/knowledgebase/1255 Reducing the amount of solution data stored in a model]
* [https://www.comsol.com/support/knowledgebase/830 Handling large models]

=== Blog-style articles ===
* [https://www.comsol.com/blogs/the-power-of-the-batch-sweep/ The Power of the Batch Sweep]
* [https://www.comsol.com/blogs/how-to-run-simulations-in-batch-mode-from-the-command-line/ How to Run Simulations in Batch Mode from the Command Line]

HPC/Applications/comsol

2025-10-03T15:30:45Z

Stern: /* Native desktop version — License configuration */

== Introduction ==
The [http://www.comsol.com/products/multiphysics/ '''COMSOL Multiphysics'''] application suite
is available for Mac, Windows, and Linux.

== Access ==
[[Image:HPC Comsol licensed modules.png|right|450px|thumb|COMSOL features licensed at CNM as of 2024-10.]]
=== COMSOL Multiphysics ===
Access to '''COMSOL Multiphysics''', the main application with both a graphical user interface and batch computing capabilities, will be granted '''''upon request''''' .

You must be:
* An '''''Argonne employee'''''  in one of the following roles:
** '''''NST staff member'''''  (including PostDocs, joint appointments, and students at NST).
** '''''CNM Facility User'''''  (including ''non-NST Argonne employees''), with an active proposal that has COMSOL in scope. Please see [[HPC/FAQ#Access_and_proposals]].

The set of available COMSOL modules that we have licensed is shown at right.

* The native desktop version can be installed only on ''Argonne-owned'' computers (laptop or desktop). Please contact [mailto:help@anl.gov?Subject=COMSOL%20license%20access%20request CNM support]. Make sure to review [http://www.comsol.com/support/updates/ Product Updates].
* To run COMSOL on our HPC system ''Carbon'', as GUI or as batch computing jobs, your proposal must have ''Carbon'' access in scope.

=== COMSOL Server ===
''CNM Facility Users'' who are not Argonne employees are not eligible to access
the full '''COMSOL Multiphysics''' product, in compliance with its licensing terms and conditions.
However, the [https://www.comsol.com/comsol-server '''COMSOL Server'''] product offers an alternative
albeit more narrow means of access.

All users, including '''''non-Argonne users''''' may be able to run, over a web interface in their browser,
pre-parameterized "fixed-model" simulations that were made with the '''COMSOL Application Builder''' (requires Windows).

To inquire about using COMSOL server applications,
please contact us.

We will review possible purchase paths for COMSOL Server, and the ''considerable effort'' likely required by users and the CNM to develop and deploy a COMSOL application.

== Support ==
* For questions on how to run or start COMSOL:
*: Submit an [[HPC/Support | '''IT support request''']]
* For questions related to most COMSOL data or scripts, contact COMSOL support at:
*: [mailto:support@comsol.com '''support'''@comsol.com]
* '''Exception:''' If your support question might contain data that is subject to '''US Export Control''' regulations, contact instead:
*: [mailto:us-export-control@us.comsol.com '''us-export-control'''@'''us'''.comsol.com]

== The modes to run COMSOL ==
There are several ways of running COMSOL.

=== Running the desktop application ===
To edit models and analyze results, use the COMSOL graphical user interface (GUI). You can run it in several ways, each explained in more detail in the following sections.

# Run on Carbon, display on your desktop using the [http://en.wikipedia.org/wiki/X_Window_System X Window system (X11)].
#* Requires an ''X11 Server'' application running on your client machine, and SSH X11 forwarding.
#* The X11 protocol can be slow over LAN.
# Run on Carbon within a [[HPC/Network Access/Virtual Desktop | virtual desktop environment (VNC)]].
#* Requires a VNC client on your machine.
#* It can be confusing where VNC and the app runs.
#* The virtual desktop presents a (deliberatly) sparse windowing environment – Copy&Paste can be difficult to use.
# Run on your own computer, remotely checking out Carbon's license.
#* Requires a full installation of COMSOL on your desktoo or laptop computer.
#* You will require access to the COMSOL license server running on Carbon, either through VPN (fairly easy) or SSH tunneling (can be intimidating for first-time users). See below for details.

=== Running a batch calculation ===
When the computational stage of models takes longer than a few minutes to complete, it will be beneficial to submit save the model file and run it in parallel on Carbon.
See the [[#Batch jobs|section below]].

== Use with X11 ==
For casual use, and with [[HPC/Network Access/SSH Tunnel Setup on Linux and MacOS | X11 tunneling to Carbon]] set up, simply add to your <code>~/.bashrc</code> file on Carbon:
module load comsol

Then, from the command line on a Carbon login node, start the COMSOL GUI:
comsol &

License configuration is automatic on Carbon.

Comsol will use a number of cores to run calculations multithreaded by default. (On Carbon, COMSOL_NUM_THREADS is set.)

'''Quit the application''' as soon as you are done. Only one user can use Comsol at a time.

== Use with VNC ==
See general [[HPC/Network Access/Virtual Desktop | instructions on using VNC]], and the COMSOL example therein.

== Native desktop version ==

To get the fastest response from the COMSOL GUI, run a native COMSOL version on your desktop machine or laptop, called ''client machine'' below.

Enter license details either in the COMSOL Setup configuration dialog.

== Native desktop version — License configuration ==
A client's COMSOL license needs to be configured depending on how the client reaches CNM network resources. Follow the appropriate section:

* [[#License configuration when inside CNM or on CNM VPN|License configuration when '''inside CNM''' or on '''CNM VPN''']]
: Requires that the user's machine is logically within the network of CNM/NST, either by a wired/auth WiFi connection in the building or by VPN under a CNM/NST profile.
* [[#License configuration when outside CNM – requires SSH Tunneling|License configuration when '''outside CNM''' – requires SSH Tunneling]]
: ''Non-NST staff'' usually have a VPN profile that places the machine logically [[#License configuration when outside CNM – requires SSH Tunneling|outside of NST's network.]]

=== License configuration when inside CNM or on CNM VPN ===
==== Ensure correct DNS search domains ====
* Inspect the client machine's network configuration.
* Ensure that '''two''' entries are listed as DNS Search Domains for the current network profile:
nst.anl.gov
cnm.anl.gov
: (The order of entries does not matter for the purposes considered here.)
* If the client machine is ''wired to CNM networks'', the domains are normally provided correctly by DHCP.
* If the client machine is ''connected over VPN'', you may need edit the respective network profile.
:* On a Mac, find the VPN configuration details under System Preferences, Network, Advanced… . You may need to unlock the Network panel first before you can make changes under "Advanced…". It appears that those changes are being saved and will be activated automatically the next time that this VPN session gets started.
: [[Image:VNC_add_cnm_to_DNS_1.png|400px]] [[Image:VNC_add_cnm_to_DNS_2.png|400px]]
:* On Windows or Linux machines, take ''analogous'' steps, using the platform's own configuration tools.

* Exclude license manager components from client desktop installation. (Those are all handled server-side.)

==== License format ====
* Floating Network License (FNL), "Three-server redundancy"

==== License servers ====
* On client machines, while running the COMSOL installer or license configuration, specify license servers and ports as follows:
1718 clicense1
1718 clicense2
1718 clicense3
: '''Note:''' Use ''short host names only''. (Technical Reason: The license must be usable both on Carbon's internal network as well as on divisional networks).
1718 clicense1<strike>.cnm.anl.gov</strike>
1718 clicense2<strike>.cnm.anl.gov</strike>
1718 clicense3<strike>.cnm.anl.gov</strike>

'''Advanced:''' A client machine's COMSOL license file <code><COMSOLDIR>/Multiphysics/license/license.dat</code> should look as follows:
<source lang="make">
SERVER clicense1 ANY 1718
SERVER clicense2 ANY 1718
SERVER clicense3 ANY 1718
USE_SERVER
</source>

<code><COMSOLDIR></code> is located by default at:
* '''Windows''': C:\Program files\COMSOL\COMSOL''nn''
* '''Linux''': /usr/local/comsol''nn''
* '''Mac''': /Applications/COMSOL''nn''
("''nn''" is the application's abbreviated version number.)

==== Network Troubleshooting ====
When using the license access model descibed here (as opposed to tunneled over SSH), ensure the following:
* Verify that short host names can be resolved to IP addresses. Either one of the following commands should produce sensible output:
host clicense1
: or:
ping -c 3 clicense1
* Verify that ''both license ports on the same server'' can be reached from your client machine:
telnet '''clicense1''' 1718
telnet '''clicense1''' 1719
: or:
telnet '''clicense2''' 1718
telnet '''clicense2''' 1719
: A ''successful'' connection looks like:
Trying 146.139.x.y...
Connected to clicense1.cnm.anl.gov.
Escape character is '^]'.
:: Do not enter any data. Instead, close each connection by hitting <code>Ctrl</code><code>]</code> (closing square bracket), followed by <code>Ctrl</code><code>D</code>.
: A ''failed'' connection gives one of these errors:
telnet: connect to address 146.139.x.y: Operation timed out
: or:
telnet: connect to address 146.139.x.y: Connection refused
:: In these cases, ask that your IP firewall configuration be reviewed and changed.

=== License configuration when outside CNM – requires SSH Tunneling ===
If you're comfortable with ssh and tunneling, set up client licensing as follows:

==== Configure COMSOL ====
The following needs to be done just once, typically during COMSOL installation:

In the COMSOL Installer, or in COMSOL Preferences, enter:
* '''License format:''' Floating Network License (FNL), "Port number and Hostname"
:: "Three-server redundancy" cannot be used because is cannot (easily) be accessed over SSH. Configure with just one server, and in rare tweak SSH to tunnel to the other server.
* '''Server:''' <code>127.0.0.1</code>
:: Do not use "localhost", as COMSOL interprets this incorrectly to mean that your machine ''is'' a license server.

* '''Port:''' <code>1718</code>


==== Provide server connection ====
This step needs to be done prior to running COMSOL in your desktop session:
* Connect to <code>mega</code> with the following ssh command:
ssh -L 1718:'''clicense1''':1718 -L 1719:'''clicense1''':1719 mega.cnm.anl.gov
* Leave the command running between invocations of COMSOL.

==== Run COMSOL ====
* Ensure your connection to "mega" is still running. If not, restart it.
* Start the COMSOL application.

** If you get an error that the number of licenses is "exhausted", that is confirmation that you correctly reached the license server, though other users are using licenses.
** If the license access attempt takes a long time or hangs, the service might temporarily run on a secondary server. In this case, stop the mega session and restart it, connecting to the alternate license server:
ssh -L 1718:clicense'''2''':1718 -L 1719:clicense'''2''':1719 mega.cnm.anl.gov

== License Troubleshooting ==
The following are typical error messages.
=== Could not obtain license for COMSOL Multiphysics GUI ===
* Error details:
License error: -4.
'''Licensed number of users already reached.'''
Feature: COMSOLGUI
License path: /Users/stern/Applications/COMSOL52a/license/license.dat:
FlexNet Licensing error:-4,132
* Reason: The connection to the license server succeeded, but the license token(s) you requested are in use.
* Solution: Wait a bit, or [http://carbonmgmt/ganglia/lic.php identify who's using COMSOL] and politely ask the user when they might be finished with their work.

=== License server machine is down or not responding ===
* Error details:
License error: -96.
See the system administrator about starting the license server system, or
make sure you're referring to the right host (see LM_LICENSE_FILE).
Feature: SERIAL
Hostname: clicense3
License path: /Users/stern/Applications/COMSOL52a/license/license.dat:
FlexNet Licensing error:-96,7
* Reason: The license servers cannot be reached at all. The <code>ping</code> or <code>host</code> tests mentioned above fail.
* Solution: Review the client license file and DNS lookup domains.

=== License file does not support this version ===
* Error details:
Feature: SERIAL
'''Application version > License version: 5.21 > 3.5'''
License path: /Users/stern/Applications/COMSOL52a/license/license.dat:
FlexNet Licensing error:-21,126
* Reason: Only the first of two steps for connecting to the license servers succeeds, which may happen when the redundant license service runs on the alternate server instead of the one you're connected to.
* Solution: When using SSH, try <code>clicense2</code> instead of <code>clicense1</code>, or vice versa. Revisit your <code>~/.ssh/config</code> file or use the alternate command line as shown in the SSH section above.

== Multiple CPU cores ==
Ways to start multi-core (from http://www.comsol.com/support/knowledgebase/1096/)
comsol -np 8

export COMSOL_NUM_THREADS=8 # automatically provided on ''Carbon''
comsol
* COMSOL does not benefit from hyperthreading (oversubscribing the physical core count).

== Batch jobs ==

* http://www.comsol.com/support/knowledgebase/1001/ .
* [[Media:HPC 2012-05-16 Comsol parallel usage.pdf]] (Modified from http://www.comsol.com/support/knowledgebase/1001/files/4027/cluster_install_linux_42a.pptx – Appendix slide, PBS)
* '''Note:''' MPD is not supported on Carbon (I patched the main comsol script), and no longer necessary at all in Comsol-4.3.

=== Sample Job file ===
See:
$COMSOL_HOME/sample.job

== Important COMSOL Support Articles ==
* Toplevel suport page: https://www.comsol.com/support

=== KB Articles ===
* [https://www.comsol.com/support/knowledgebase/1001 Running COMSOL® in parallel on clusters]
* [https://www.comsol.com/support/knowledgebase/1250 Running parametric sweeps, batch sweeps, and cluster sweeps from the command line]
* [https://www.comsol.com/support/knowledgebase/1255 Reducing the amount of solution data stored in a model]
* [https://www.comsol.com/support/knowledgebase/830 Handling large models]

=== Blog-style articles ===
* [https://www.comsol.com/blogs/the-power-of-the-batch-sweep/ The Power of the Batch Sweep]
* [https://www.comsol.com/blogs/how-to-run-simulations-in-batch-mode-from-the-command-line/ How to Run Simulations in Batch Mode from the Command Line]

HPC/Applications/comsol

2025-10-03T15:29:12Z

Stern: /* Native desktop version — License configuration */

== Introduction ==
The [http://www.comsol.com/products/multiphysics/ '''COMSOL Multiphysics'''] application suite
is available for Mac, Windows, and Linux.

== Access ==
[[Image:HPC Comsol licensed modules.png|right|450px|thumb|COMSOL features licensed at CNM as of 2024-10.]]
=== COMSOL Multiphysics ===
Access to '''COMSOL Multiphysics''', the main application with both a graphical user interface and batch computing capabilities, will be granted '''''upon request''''' .

You must be:
* An '''''Argonne employee'''''  in one of the following roles:
** '''''NST staff member'''''  (including PostDocs, joint appointments, and students at NST).
** '''''CNM Facility User'''''  (including ''non-NST Argonne employees''), with an active proposal that has COMSOL in scope. Please see [[HPC/FAQ#Access_and_proposals]].

The set of available COMSOL modules that we have licensed is shown at right.

* The native desktop version can be installed only on ''Argonne-owned'' computers (laptop or desktop). Please contact [mailto:help@anl.gov?Subject=COMSOL%20license%20access%20request CNM support]. Make sure to review [http://www.comsol.com/support/updates/ Product Updates].
* To run COMSOL on our HPC system ''Carbon'', as GUI or as batch computing jobs, your proposal must have ''Carbon'' access in scope.

=== COMSOL Server ===
''CNM Facility Users'' who are not Argonne employees are not eligible to access
the full '''COMSOL Multiphysics''' product, in compliance with its licensing terms and conditions.
However, the [https://www.comsol.com/comsol-server '''COMSOL Server'''] product offers an alternative
albeit more narrow means of access.

All users, including '''''non-Argonne users''''' may be able to run, over a web interface in their browser,
pre-parameterized "fixed-model" simulations that were made with the '''COMSOL Application Builder''' (requires Windows).

To inquire about using COMSOL server applications,
please contact us.

We will review possible purchase paths for COMSOL Server, and the ''considerable effort'' likely required by users and the CNM to develop and deploy a COMSOL application.

== Support ==
* For questions on how to run or start COMSOL:
*: Submit an [[HPC/Support | '''IT support request''']]
* For questions related to most COMSOL data or scripts, contact COMSOL support at:
*: [mailto:support@comsol.com '''support'''@comsol.com]
* '''Exception:''' If your support question might contain data that is subject to '''US Export Control''' regulations, contact instead:
*: [mailto:us-export-control@us.comsol.com '''us-export-control'''@'''us'''.comsol.com]

== The modes to run COMSOL ==
There are several ways of running COMSOL.

=== Running the desktop application ===
To edit models and analyze results, use the COMSOL graphical user interface (GUI). You can run it in several ways, each explained in more detail in the following sections.

# Run on Carbon, display on your desktop using the [http://en.wikipedia.org/wiki/X_Window_System X Window system (X11)].
#* Requires an ''X11 Server'' application running on your client machine, and SSH X11 forwarding.
#* The X11 protocol can be slow over LAN.
# Run on Carbon within a [[HPC/Network Access/Virtual Desktop | virtual desktop environment (VNC)]].
#* Requires a VNC client on your machine.
#* It can be confusing where VNC and the app runs.
#* The virtual desktop presents a (deliberatly) sparse windowing environment – Copy&Paste can be difficult to use.
# Run on your own computer, remotely checking out Carbon's license.
#* Requires a full installation of COMSOL on your desktoo or laptop computer.
#* You will require access to the COMSOL license server running on Carbon, either through VPN (fairly easy) or SSH tunneling (can be intimidating for first-time users). See below for details.

=== Running a batch calculation ===
When the computational stage of models takes longer than a few minutes to complete, it will be beneficial to submit save the model file and run it in parallel on Carbon.
See the [[#Batch jobs|section below]].

== Use with X11 ==
For casual use, and with [[HPC/Network Access/SSH Tunnel Setup on Linux and MacOS | X11 tunneling to Carbon]] set up, simply add to your <code>~/.bashrc</code> file on Carbon:
module load comsol

Then, from the command line on a Carbon login node, start the COMSOL GUI:
comsol &

License configuration is automatic on Carbon.

Comsol will use a number of cores to run calculations multithreaded by default. (On Carbon, COMSOL_NUM_THREADS is set.)

'''Quit the application''' as soon as you are done. Only one user can use Comsol at a time.

== Use with VNC ==
See general [[HPC/Network Access/Virtual Desktop | instructions on using VNC]], and the COMSOL example therein.

== Native desktop version ==

To get the fastest response from the COMSOL GUI, run a native COMSOL version on your desktop machine or laptop, called ''client machine'' below.

Enter license details either in the COMSOL Setup configuration dialog.

== Native desktop version — License configuration ==
A client's COMSOL license needs to be configured depending on how the client reaches CNM network resources. Follow the appropriate section:

* [[#License configuration when inside CNM or on CNM VPN|License configuration when '''inside CNM''' or on '''CNM VPN''']]
* [[#License configuration when outside CNM – requires SSH Tunneling|License configuration when '''outside CNM''' – requires SSH Tunneling]]

=== License configuration when inside CNM or on CNM VPN ===
Requires that the user's machine is logically within the network of CNM/NST, either by a wired/auth WiFi connection in the building or by VPN under a CNM/NST profile.

''Non-NST staff'' usually have a VPN profile that places the machine logically [[#License configuration when outside CNM – requires SSH Tunneling|outside of NST's network.]]

==== Ensure correct DNS search domains ====
* Inspect the client machine's network configuration.
* Ensure that '''two''' entries are listed as DNS Search Domains for the current network profile:
nst.anl.gov
cnm.anl.gov
: (The order of entries does not matter for the purposes considered here.)
* If the client machine is ''wired to CNM networks'', the domains are normally provided correctly by DHCP.
* If the client machine is ''connected over VPN'', you may need edit the respective network profile.
:* On a Mac, find the VPN configuration details under System Preferences, Network, Advanced… . You may need to unlock the Network panel first before you can make changes under "Advanced…". It appears that those changes are being saved and will be activated automatically the next time that this VPN session gets started.
: [[Image:VNC_add_cnm_to_DNS_1.png|400px]] [[Image:VNC_add_cnm_to_DNS_2.png|400px]]
:* On Windows or Linux machines, take ''analogous'' steps, using the platform's own configuration tools.

* Exclude license manager components from client desktop installation. (Those are all handled server-side.)

==== License format ====
* Floating Network License (FNL), "Three-server redundancy"

==== License servers ====
* On client machines, while running the COMSOL installer or license configuration, specify license servers and ports as follows:
1718 clicense1
1718 clicense2
1718 clicense3
: '''Note:''' Use ''short host names only''. (Technical Reason: The license must be usable both on Carbon's internal network as well as on divisional networks).
1718 clicense1<strike>.cnm.anl.gov</strike>
1718 clicense2<strike>.cnm.anl.gov</strike>
1718 clicense3<strike>.cnm.anl.gov</strike>

'''Advanced:''' A client machine's COMSOL license file <code><COMSOLDIR>/Multiphysics/license/license.dat</code> should look as follows:
<source lang="make">
SERVER clicense1 ANY 1718
SERVER clicense2 ANY 1718
SERVER clicense3 ANY 1718
USE_SERVER
</source>

<code><COMSOLDIR></code> is located by default at:
* '''Windows''': C:\Program files\COMSOL\COMSOL''nn''
* '''Linux''': /usr/local/comsol''nn''
* '''Mac''': /Applications/COMSOL''nn''
("''nn''" is the application's abbreviated version number.)

==== Network Troubleshooting ====
When using the license access model descibed here (as opposed to tunneled over SSH), ensure the following:
* Verify that short host names can be resolved to IP addresses. Either one of the following commands should produce sensible output:
host clicense1
: or:
ping -c 3 clicense1
* Verify that ''both license ports on the same server'' can be reached from your client machine:
telnet '''clicense1''' 1718
telnet '''clicense1''' 1719
: or:
telnet '''clicense2''' 1718
telnet '''clicense2''' 1719
: A ''successful'' connection looks like:
Trying 146.139.x.y...
Connected to clicense1.cnm.anl.gov.
Escape character is '^]'.
:: Do not enter any data. Instead, close each connection by hitting <code>Ctrl</code><code>]</code> (closing square bracket), followed by <code>Ctrl</code><code>D</code>.
: A ''failed'' connection gives one of these errors:
telnet: connect to address 146.139.x.y: Operation timed out
: or:
telnet: connect to address 146.139.x.y: Connection refused
:: In these cases, ask that your IP firewall configuration be reviewed and changed.

=== License configuration when outside CNM – requires SSH Tunneling ===
If you're comfortable with ssh and tunneling, set up client licensing as follows:

==== Configure COMSOL ====
The following needs to be done just once, typically during COMSOL installation:

In the COMSOL Installer, or in COMSOL Preferences, enter:
* '''License format:''' Floating Network License (FNL), "Port number and Hostname"
:: "Three-server redundancy" cannot be used because is cannot (easily) be accessed over SSH. Configure with just one server, and in rare tweak SSH to tunnel to the other server.
* '''Server:''' <code>127.0.0.1</code>
:: Do not use "localhost", as COMSOL interprets this incorrectly to mean that your machine ''is'' a license server.

* '''Port:''' <code>1718</code>


==== Provide server connection ====
This step needs to be done prior to running COMSOL in your desktop session:
* Connect to <code>mega</code> with the following ssh command:
ssh -L 1718:'''clicense1''':1718 -L 1719:'''clicense1''':1719 mega.cnm.anl.gov
* Leave the command running between invocations of COMSOL.

==== Run COMSOL ====
* Ensure your connection to "mega" is still running. If not, restart it.
* Start the COMSOL application.

** If you get an error that the number of licenses is "exhausted", that is confirmation that you correctly reached the license server, though other users are using licenses.
** If the license access attempt takes a long time or hangs, the service might temporarily run on a secondary server. In this case, stop the mega session and restart it, connecting to the alternate license server:
ssh -L 1718:clicense'''2''':1718 -L 1719:clicense'''2''':1719 mega.cnm.anl.gov

== License Troubleshooting ==
The following are typical error messages.
=== Could not obtain license for COMSOL Multiphysics GUI ===
* Error details:
License error: -4.
'''Licensed number of users already reached.'''
Feature: COMSOLGUI
License path: /Users/stern/Applications/COMSOL52a/license/license.dat:
FlexNet Licensing error:-4,132
* Reason: The connection to the license server succeeded, but the license token(s) you requested are in use.
* Solution: Wait a bit, or [http://carbonmgmt/ganglia/lic.php identify who's using COMSOL] and politely ask the user when they might be finished with their work.

=== License server machine is down or not responding ===
* Error details:
License error: -96.
See the system administrator about starting the license server system, or
make sure you're referring to the right host (see LM_LICENSE_FILE).
Feature: SERIAL
Hostname: clicense3
License path: /Users/stern/Applications/COMSOL52a/license/license.dat:
FlexNet Licensing error:-96,7
* Reason: The license servers cannot be reached at all. The <code>ping</code> or <code>host</code> tests mentioned above fail.
* Solution: Review the client license file and DNS lookup domains.

=== License file does not support this version ===
* Error details:
Feature: SERIAL
'''Application version > License version: 5.21 > 3.5'''
License path: /Users/stern/Applications/COMSOL52a/license/license.dat:
FlexNet Licensing error:-21,126
* Reason: Only the first of two steps for connecting to the license servers succeeds, which may happen when the redundant license service runs on the alternate server instead of the one you're connected to.
* Solution: When using SSH, try <code>clicense2</code> instead of <code>clicense1</code>, or vice versa. Revisit your <code>~/.ssh/config</code> file or use the alternate command line as shown in the SSH section above.

== Multiple CPU cores ==
Ways to start multi-core (from http://www.comsol.com/support/knowledgebase/1096/)
comsol -np 8

export COMSOL_NUM_THREADS=8 # automatically provided on ''Carbon''
comsol
* COMSOL does not benefit from hyperthreading (oversubscribing the physical core count).

== Batch jobs ==

* http://www.comsol.com/support/knowledgebase/1001/ .
* [[Media:HPC 2012-05-16 Comsol parallel usage.pdf]] (Modified from http://www.comsol.com/support/knowledgebase/1001/files/4027/cluster_install_linux_42a.pptx – Appendix slide, PBS)
* '''Note:''' MPD is not supported on Carbon (I patched the main comsol script), and no longer necessary at all in Comsol-4.3.

=== Sample Job file ===
See:
$COMSOL_HOME/sample.job

== Important COMSOL Support Articles ==
* Toplevel suport page: https://www.comsol.com/support

=== KB Articles ===
* [https://www.comsol.com/support/knowledgebase/1001 Running COMSOL® in parallel on clusters]
* [https://www.comsol.com/support/knowledgebase/1250 Running parametric sweeps, batch sweeps, and cluster sweeps from the command line]
* [https://www.comsol.com/support/knowledgebase/1255 Reducing the amount of solution data stored in a model]
* [https://www.comsol.com/support/knowledgebase/830 Handling large models]

=== Blog-style articles ===
* [https://www.comsol.com/blogs/the-power-of-the-batch-sweep/ The Power of the Batch Sweep]
* [https://www.comsol.com/blogs/how-to-run-simulations-in-batch-mode-from-the-command-line/ How to Run Simulations in Batch Mode from the Command Line]

HPC/Applications/comsol

2025-10-03T15:21:31Z

Stern: /* Native desktop version — License configuration */

== Introduction ==
The [http://www.comsol.com/products/multiphysics/ '''COMSOL Multiphysics'''] application suite
is available for Mac, Windows, and Linux.

== Access ==
[[Image:HPC Comsol licensed modules.png|right|450px|thumb|COMSOL features licensed at CNM as of 2024-10.]]
=== COMSOL Multiphysics ===
Access to '''COMSOL Multiphysics''', the main application with both a graphical user interface and batch computing capabilities, will be granted '''''upon request''''' .

You must be:
* An '''''Argonne employee'''''  in one of the following roles:
** '''''NST staff member'''''  (including PostDocs, joint appointments, and students at NST).
** '''''CNM Facility User'''''  (including ''non-NST Argonne employees''), with an active proposal that has COMSOL in scope. Please see [[HPC/FAQ#Access_and_proposals]].

The set of available COMSOL modules that we have licensed is shown at right.

* The native desktop version can be installed only on ''Argonne-owned'' computers (laptop or desktop). Please contact [mailto:help@anl.gov?Subject=COMSOL%20license%20access%20request CNM support]. Make sure to review [http://www.comsol.com/support/updates/ Product Updates].
* To run COMSOL on our HPC system ''Carbon'', as GUI or as batch computing jobs, your proposal must have ''Carbon'' access in scope.

=== COMSOL Server ===
''CNM Facility Users'' who are not Argonne employees are not eligible to access
the full '''COMSOL Multiphysics''' product, in compliance with its licensing terms and conditions.
However, the [https://www.comsol.com/comsol-server '''COMSOL Server'''] product offers an alternative
albeit more narrow means of access.

All users, including '''''non-Argonne users''''' may be able to run, over a web interface in their browser,
pre-parameterized "fixed-model" simulations that were made with the '''COMSOL Application Builder''' (requires Windows).

To inquire about using COMSOL server applications,
please contact us.

We will review possible purchase paths for COMSOL Server, and the ''considerable effort'' likely required by users and the CNM to develop and deploy a COMSOL application.

== Support ==
* For questions on how to run or start COMSOL:
*: Submit an [[HPC/Support | '''IT support request''']]
* For questions related to most COMSOL data or scripts, contact COMSOL support at:
*: [mailto:support@comsol.com '''support'''@comsol.com]
* '''Exception:''' If your support question might contain data that is subject to '''US Export Control''' regulations, contact instead:
*: [mailto:us-export-control@us.comsol.com '''us-export-control'''@'''us'''.comsol.com]

== The modes to run COMSOL ==
There are several ways of running COMSOL.

=== Running the desktop application ===
To edit models and analyze results, use the COMSOL graphical user interface (GUI). You can run it in several ways, each explained in more detail in the following sections.

# Run on Carbon, display on your desktop using the [http://en.wikipedia.org/wiki/X_Window_System X Window system (X11)].
#* Requires an ''X11 Server'' application running on your client machine, and SSH X11 forwarding.
#* The X11 protocol can be slow over LAN.
# Run on Carbon within a [[HPC/Network Access/Virtual Desktop | virtual desktop environment (VNC)]].
#* Requires a VNC client on your machine.
#* It can be confusing where VNC and the app runs.
#* The virtual desktop presents a (deliberatly) sparse windowing environment – Copy&Paste can be difficult to use.
# Run on your own computer, remotely checking out Carbon's license.
#* Requires a full installation of COMSOL on your desktoo or laptop computer.
#* You will require access to the COMSOL license server running on Carbon, either through VPN (fairly easy) or SSH tunneling (can be intimidating for first-time users). See below for details.

=== Running a batch calculation ===
When the computational stage of models takes longer than a few minutes to complete, it will be beneficial to submit save the model file and run it in parallel on Carbon.
See the [[#Batch jobs|section below]].

== Use with X11 ==
For casual use, and with [[HPC/Network Access/SSH Tunnel Setup on Linux and MacOS | X11 tunneling to Carbon]] set up, simply add to your <code>~/.bashrc</code> file on Carbon:
module load comsol

Then, from the command line on a Carbon login node, start the COMSOL GUI:
comsol &

License configuration is automatic on Carbon.

Comsol will use a number of cores to run calculations multithreaded by default. (On Carbon, COMSOL_NUM_THREADS is set.)

'''Quit the application''' as soon as you are done. Only one user can use Comsol at a time.

== Use with VNC ==
See general [[HPC/Network Access/Virtual Desktop | instructions on using VNC]], and the COMSOL example therein.

== Native desktop version ==

To get the fastest response from the COMSOL GUI, run a native COMSOL version on your desktop machine or laptop, called ''client machine'' below.

Enter license details either in the COMSOL Setup configuration dialog.

== Native desktop version — License configuration ==
A client's COMSOL license needs to be configured depending on how the client reaches CNM network resources. Follow the appropriate section:

* [[#License configuration when inside CNM or on VPN under a CNM/NST staff member|License configuration when '''inside CNM''' or on '''VPN''' under a '''CNM/NST staff member''']]
* [[#License configuration when outside CNM – requires SSH Tunneling|License configuration when '''outside CNM''' – requires SSH Tunneling]]

=== License configuration when inside CNM or on VPN under a CNM/NST staff member ===
==== Ensure correct DNS search domains ====

* Inspect the client machine's network configuration.
* Ensure that '''two''' entries are listed as DNS Search Domains for the current network profile:
nst.anl.gov
cnm.anl.gov
: (The order of entries does not matter for the purposes considered here.)
* If the client machine is ''wired to CNM networks'', the domains are normally provided correctly by DHCP.
* If the client machine is ''connected over VPN'', you may need edit the respective network profile.
:* On a Mac, find the VPN configuration details under System Preferences, Network, Advanced… . You may need to unlock the Network panel first before you can make changes under "Advanced…". It appears that those changes are being saved and will be activated automatically the next time that this VPN session gets started.
: [[Image:VNC_add_cnm_to_DNS_1.png|400px]] [[Image:VNC_add_cnm_to_DNS_2.png|400px]]
:* On Windows or Linux machines, take ''analogous'' steps, using the platform's own configuration tools.

* Exclude license manager components from client desktop installation. (Those are all handled server-side.)

==== License format ====
* Floating Network License (FNL), "Three-server redundancy"

==== License servers ====
* On client machines, while running the COMSOL installer or license configuration, specify license servers and ports as follows:
1718 clicense1
1718 clicense2
1718 clicense3
: '''Note:''' Use ''short host names only''. (Technical Reason: The license must be usable both on Carbon's internal network as well as on divisional networks).
1718 clicense1<strike>.cnm.anl.gov</strike>
1718 clicense2<strike>.cnm.anl.gov</strike>
1718 clicense3<strike>.cnm.anl.gov</strike>

'''Advanced:''' A client machine's COMSOL license file <code><COMSOLDIR>/Multiphysics/license/license.dat</code> should look as follows:
<source lang="make">
SERVER clicense1 ANY 1718
SERVER clicense2 ANY 1718
SERVER clicense3 ANY 1718
USE_SERVER
</source>

<code><COMSOLDIR></code> is located by default at:
* '''Windows''': C:\Program files\COMSOL\COMSOL''nn''
* '''Linux''': /usr/local/comsol''nn''
* '''Mac''': /Applications/COMSOL''nn''
("''nn''" is the application's abbreviated version number.)

==== Network Troubleshooting ====
When using the license access model descibed here (as opposed to tunneled over SSH), ensure the following:
* Verify that short host names can be resolved to IP addresses. Either one of the following commands should produce sensible output:
host clicense1
: or:
ping -c 3 clicense1
* Verify that ''both license ports on the same server'' can be reached from your client machine:
telnet '''clicense1''' 1718
telnet '''clicense1''' 1719
: or:
telnet '''clicense2''' 1718
telnet '''clicense2''' 1719
: A ''successful'' connection looks like:
Trying 146.139.x.y...
Connected to clicense1.cnm.anl.gov.
Escape character is '^]'.
:: Do not enter any data. Instead, close each connection by hitting <code>Ctrl</code><code>]</code> (closing square bracket), followed by <code>Ctrl</code><code>D</code>.
: A ''failed'' connection gives one of these errors:
telnet: connect to address 146.139.x.y: Operation timed out
: or:
telnet: connect to address 146.139.x.y: Connection refused
:: In these cases, ask that your IP firewall configuration be reviewed and changed.

=== License configuration when outside CNM – requires SSH Tunneling ===
If you're comfortable with ssh and tunneling, set up client licensing as follows:

==== Configure COMSOL ====
The following needs to be done just once, typically during COMSOL installation:

In the COMSOL Installer, or in COMSOL Preferences, enter:
* '''License format:''' Floating Network License (FNL), "Port number and Hostname"
:: "Three-server redundancy" cannot be used because is cannot (easily) be accessed over SSH. Configure with just one server, and in rare tweak SSH to tunnel to the other server.
* '''Server:''' <code>127.0.0.1</code>
:: Do not use "localhost", as COMSOL interprets this incorrectly to mean that your machine ''is'' a license server.

* '''Port:''' <code>1718</code>


==== Provide server connection ====
This step needs to be done prior to running COMSOL in your desktop session:
* Connect to <code>mega</code> with the following ssh command:
ssh -L 1718:'''clicense1''':1718 -L 1719:'''clicense1''':1719 mega.cnm.anl.gov
* Leave the command running between invocations of COMSOL.

==== Run COMSOL ====
* Ensure your connection to "mega" is still running. If not, restart it.
* Start the COMSOL application.

** If you get an error that the number of licenses is "exhausted", that is confirmation that you correctly reached the license server, though other users are using licenses.
** If the license access attempt takes a long time or hangs, the service might temporarily run on a secondary server. In this case, stop the mega session and restart it, connecting to the alternate license server:
ssh -L 1718:clicense'''2''':1718 -L 1719:clicense'''2''':1719 mega.cnm.anl.gov

== License Troubleshooting ==
The following are typical error messages.
=== Could not obtain license for COMSOL Multiphysics GUI ===
* Error details:
License error: -4.
'''Licensed number of users already reached.'''
Feature: COMSOLGUI
License path: /Users/stern/Applications/COMSOL52a/license/license.dat:
FlexNet Licensing error:-4,132
* Reason: The connection to the license server succeeded, but the license token(s) you requested are in use.
* Solution: Wait a bit, or [http://carbonmgmt/ganglia/lic.php identify who's using COMSOL] and politely ask the user when they might be finished with their work.

=== License server machine is down or not responding ===
* Error details:
License error: -96.
See the system administrator about starting the license server system, or
make sure you're referring to the right host (see LM_LICENSE_FILE).
Feature: SERIAL
Hostname: clicense3
License path: /Users/stern/Applications/COMSOL52a/license/license.dat:
FlexNet Licensing error:-96,7
* Reason: The license servers cannot be reached at all. The <code>ping</code> or <code>host</code> tests mentioned above fail.
* Solution: Review the client license file and DNS lookup domains.

=== License file does not support this version ===
* Error details:
Feature: SERIAL
'''Application version > License version: 5.21 > 3.5'''
License path: /Users/stern/Applications/COMSOL52a/license/license.dat:
FlexNet Licensing error:-21,126
* Reason: Only the first of two steps for connecting to the license servers succeeds, which may happen when the redundant license service runs on the alternate server instead of the one you're connected to.
* Solution: When using SSH, try <code>clicense2</code> instead of <code>clicense1</code>, or vice versa. Revisit your <code>~/.ssh/config</code> file or use the alternate command line as shown in the SSH section above.

== Multiple CPU cores ==
Ways to start multi-core (from http://www.comsol.com/support/knowledgebase/1096/)
comsol -np 8

export COMSOL_NUM_THREADS=8 # automatically provided on ''Carbon''
comsol
* COMSOL does not benefit from hyperthreading (oversubscribing the physical core count).

== Batch jobs ==

* http://www.comsol.com/support/knowledgebase/1001/ .
* [[Media:HPC 2012-05-16 Comsol parallel usage.pdf]] (Modified from http://www.comsol.com/support/knowledgebase/1001/files/4027/cluster_install_linux_42a.pptx – Appendix slide, PBS)
* '''Note:''' MPD is not supported on Carbon (I patched the main comsol script), and no longer necessary at all in Comsol-4.3.

=== Sample Job file ===
See:
$COMSOL_HOME/sample.job

== Important COMSOL Support Articles ==
* Toplevel suport page: https://www.comsol.com/support

=== KB Articles ===
* [https://www.comsol.com/support/knowledgebase/1001 Running COMSOL® in parallel on clusters]
* [https://www.comsol.com/support/knowledgebase/1250 Running parametric sweeps, batch sweeps, and cluster sweeps from the command line]
* [https://www.comsol.com/support/knowledgebase/1255 Reducing the amount of solution data stored in a model]
* [https://www.comsol.com/support/knowledgebase/830 Handling large models]

=== Blog-style articles ===
* [https://www.comsol.com/blogs/the-power-of-the-batch-sweep/ The Power of the Batch Sweep]
* [https://www.comsol.com/blogs/how-to-run-simulations-in-batch-mode-from-the-command-line/ How to Run Simulations in Batch Mode from the Command Line]

HPC/Applications/comsol

2025-10-03T15:14:18Z

Stern: /* License servers */

== Introduction ==
The [http://www.comsol.com/products/multiphysics/ '''COMSOL Multiphysics'''] application suite
is available for Mac, Windows, and Linux.

== Access ==
[[Image:HPC Comsol licensed modules.png|right|450px|thumb|COMSOL features licensed at CNM as of 2024-10.]]
=== COMSOL Multiphysics ===
Access to '''COMSOL Multiphysics''', the main application with both a graphical user interface and batch computing capabilities, will be granted '''''upon request''''' .

You must be:
* An '''''Argonne employee'''''  in one of the following roles:
** '''''NST staff member'''''  (including PostDocs, joint appointments, and students at NST).
** '''''CNM Facility User'''''  (including ''non-NST Argonne employees''), with an active proposal that has COMSOL in scope. Please see [[HPC/FAQ#Access_and_proposals]].

The set of available COMSOL modules that we have licensed is shown at right.

* The native desktop version can be installed only on ''Argonne-owned'' computers (laptop or desktop). Please contact [mailto:help@anl.gov?Subject=COMSOL%20license%20access%20request CNM support]. Make sure to review [http://www.comsol.com/support/updates/ Product Updates].
* To run COMSOL on our HPC system ''Carbon'', as GUI or as batch computing jobs, your proposal must have ''Carbon'' access in scope.

=== COMSOL Server ===
''CNM Facility Users'' who are not Argonne employees are not eligible to access
the full '''COMSOL Multiphysics''' product, in compliance with its licensing terms and conditions.
However, the [https://www.comsol.com/comsol-server '''COMSOL Server'''] product offers an alternative
albeit more narrow means of access.

All users, including '''''non-Argonne users''''' may be able to run, over a web interface in their browser,
pre-parameterized "fixed-model" simulations that were made with the '''COMSOL Application Builder''' (requires Windows).

To inquire about using COMSOL server applications,
please contact us.

We will review possible purchase paths for COMSOL Server, and the ''considerable effort'' likely required by users and the CNM to develop and deploy a COMSOL application.

== Support ==
* For questions on how to run or start COMSOL:
*: Submit an [[HPC/Support | '''IT support request''']]
* For questions related to most COMSOL data or scripts, contact COMSOL support at:
*: [mailto:support@comsol.com '''support'''@comsol.com]
* '''Exception:''' If your support question might contain data that is subject to '''US Export Control''' regulations, contact instead:
*: [mailto:us-export-control@us.comsol.com '''us-export-control'''@'''us'''.comsol.com]

== The modes to run COMSOL ==
There are several ways of running COMSOL.

=== Running the desktop application ===
To edit models and analyze results, use the COMSOL graphical user interface (GUI). You can run it in several ways, each explained in more detail in the following sections.

# Run on Carbon, display on your desktop using the [http://en.wikipedia.org/wiki/X_Window_System X Window system (X11)].
#* Requires an ''X11 Server'' application running on your client machine, and SSH X11 forwarding.
#* The X11 protocol can be slow over LAN.
# Run on Carbon within a [[HPC/Network Access/Virtual Desktop | virtual desktop environment (VNC)]].
#* Requires a VNC client on your machine.
#* It can be confusing where VNC and the app runs.
#* The virtual desktop presents a (deliberatly) sparse windowing environment – Copy&Paste can be difficult to use.
# Run on your own computer, remotely checking out Carbon's license.
#* Requires a full installation of COMSOL on your desktoo or laptop computer.
#* You will require access to the COMSOL license server running on Carbon, either through VPN (fairly easy) or SSH tunneling (can be intimidating for first-time users). See below for details.

=== Running a batch calculation ===
When the computational stage of models takes longer than a few minutes to complete, it will be beneficial to submit save the model file and run it in parallel on Carbon.
See the [[#Batch jobs|section below]].

== Use with X11 ==
For casual use, and with [[HPC/Network Access/SSH Tunnel Setup on Linux and MacOS | X11 tunneling to Carbon]] set up, simply add to your <code>~/.bashrc</code> file on Carbon:
module load comsol

Then, from the command line on a Carbon login node, start the COMSOL GUI:
comsol &

License configuration is automatic on Carbon.

Comsol will use a number of cores to run calculations multithreaded by default. (On Carbon, COMSOL_NUM_THREADS is set.)

'''Quit the application''' as soon as you are done. Only one user can use Comsol at a time.

== Use with VNC ==
See general [[HPC/Network Access/Virtual Desktop | instructions on using VNC]], and the COMSOL example therein.

== Native desktop version ==

To get the fastest response from the COMSOL GUI, run a native COMSOL version on your desktop machine or laptop, called ''client machine'' below.

Enter license details either in the COMSOL Setup configuration dialog.

== Native desktop version — License configuration ==
A client's COMSOL license needs to be configured depending on how the client reaches CNM network resources. Follow the appropriate section:
* [[#License configuration when inside CNM or using VPN]]
* [[#License configuration when outside CNM and using SSH Tunneling]]

=== License configuration when inside CNM or using VPN ===
==== Ensure correct DNS search domains ====

* Inspect the client machine's network configuration.
* Ensure that '''two''' entries are listed as DNS Search Domains for the current network profile:
nst.anl.gov
cnm.anl.gov
: (The order of entries does not matter for the purposes considered here.)
* If the client machine is ''wired to CNM networks'', the domains are normally provided correctly by DHCP.
* If the client machine is ''connected over VPN'', you may need edit the respective network profile.
:* On a Mac, find the VPN configuration details under System Preferences, Network, Advanced… . You may need to unlock the Network panel first before you can make changes under "Advanced…". It appears that those changes are being saved and will be activated automatically the next time that this VPN session gets started.
: [[Image:VNC_add_cnm_to_DNS_1.png|400px]] [[Image:VNC_add_cnm_to_DNS_2.png|400px]]
:* On Windows or Linux machines, take ''analogous'' steps, using the platform's own configuration tools.

* Exclude license manager components from client desktop installation. (Those are all handled server-side.)

==== License format ====
* Floating Network License (FNL), "Three-server redundancy"

==== License servers ====
* On client machines, while running the COMSOL installer or license configuration, specify license servers and ports as follows:
1718 clicense1
1718 clicense2
1718 clicense3
: '''Note:''' Use ''short host names only''. (Technical Reason: The license must be usable both on Carbon's internal network as well as on divisional networks).
1718 clicense1<strike>.cnm.anl.gov</strike>
1718 clicense2<strike>.cnm.anl.gov</strike>
1718 clicense3<strike>.cnm.anl.gov</strike>

'''Advanced:''' A client machine's COMSOL license file <code><COMSOLDIR>/Multiphysics/license/license.dat</code> should look as follows:
<source lang="make">
SERVER clicense1 ANY 1718
SERVER clicense2 ANY 1718
SERVER clicense3 ANY 1718
USE_SERVER
</source>

<code><COMSOLDIR></code> is located by default at:
* '''Windows''': C:\Program files\COMSOL\COMSOL''nn''
* '''Linux''': /usr/local/comsol''nn''
* '''Mac''': /Applications/COMSOL''nn''
("''nn''" is the application's abbreviated version number.)

==== Network Troubleshooting ====
When using the license access model descibed here (as opposed to tunneled over SSH), ensure the following:
* Verify that short host names can be resolved to IP addresses. Either one of the following commands should produce sensible output:
host clicense1
: or:
ping -c 3 clicense1
* Verify that ''both license ports on the same server'' can be reached from your client machine:
telnet '''clicense1''' 1718
telnet '''clicense1''' 1719
: or:
telnet '''clicense2''' 1718
telnet '''clicense2''' 1719
: A ''successful'' connection looks like:
Trying 146.139.x.y...
Connected to clicense1.cnm.anl.gov.
Escape character is '^]'.
:: Do not enter any data. Instead, close each connection by hitting <code>Ctrl</code><code>]</code> (closing square bracket), followed by <code>Ctrl</code><code>D</code>.
: A ''failed'' connection gives one of these errors:
telnet: connect to address 146.139.x.y: Operation timed out
: or:
telnet: connect to address 146.139.x.y: Connection refused
:: In these cases, ask that your IP firewall configuration be reviewed and changed.

=== License access when outside CNM ===
If you're comfortable with ssh and tunneling, set up client licensing as follows:

==== Configure COMSOL ====
The following needs to be done just once, typically during COMSOL installation:

In the COMSOL Installer, or in COMSOL Preferences, enter:
* '''License format:''' Floating Network License (FNL), "Port number and Hostname"
:: "Three-server redundancy" cannot be used because is cannot (easily) be accessed over SSH. Configure with just one server, and in rare tweak SSH to tunnel to the other server.
* '''Server:''' <code>127.0.0.1</code>
:: Do not use "localhost", as COMSOL interprets this incorrectly to mean that your machine ''is'' a license server.

* '''Port:''' <code>1718</code>


==== Provide server connection ====
This step needs to be done prior to running COMSOL in your desktop session:
* Connect to <code>mega</code> with the following ssh command:
ssh -L 1718:'''clicense1''':1718 -L 1719:'''clicense1''':1719 mega.cnm.anl.gov
* Leave the command running between invocations of COMSOL.

==== Run COMSOL ====
* Ensure your connection to "mega" is still running. If not, restart it.
* Start the COMSOL application.

** If you get an error that the number of licenses is "exhausted", that is confirmation that you correctly reached the license server, though other users are using licenses.
** If the license access attempt takes a long time or hangs, the service might temporarily run on a secondary server. In this case, stop the mega session and restart it, connecting to the alternate license server:
ssh -L 1718:clicense'''2''':1718 -L 1719:clicense'''2''':1719 mega.cnm.anl.gov

== License Troubleshooting ==
The following are typical error messages.
=== Could not obtain license for COMSOL Multiphysics GUI ===
* Error details:
License error: -4.
'''Licensed number of users already reached.'''
Feature: COMSOLGUI
License path: /Users/stern/Applications/COMSOL52a/license/license.dat:
FlexNet Licensing error:-4,132
* Reason: The connection to the license server succeeded, but the license token(s) you requested are in use.
* Solution: Wait a bit, or [http://carbonmgmt/ganglia/lic.php identify who's using COMSOL] and politely ask the user when they might be finished with their work.

=== License server machine is down or not responding ===
* Error details:
License error: -96.
See the system administrator about starting the license server system, or
make sure you're referring to the right host (see LM_LICENSE_FILE).
Feature: SERIAL
Hostname: clicense3
License path: /Users/stern/Applications/COMSOL52a/license/license.dat:
FlexNet Licensing error:-96,7
* Reason: The license servers cannot be reached at all. The <code>ping</code> or <code>host</code> tests mentioned above fail.
* Solution: Review the client license file and DNS lookup domains.

=== License file does not support this version ===
* Error details:
Feature: SERIAL
'''Application version > License version: 5.21 > 3.5'''
License path: /Users/stern/Applications/COMSOL52a/license/license.dat:
FlexNet Licensing error:-21,126
* Reason: Only the first of two steps for connecting to the license servers succeeds, which may happen when the redundant license service runs on the alternate server instead of the one you're connected to.
* Solution: When using SSH, try <code>clicense2</code> instead of <code>clicense1</code>, or vice versa. Revisit your <code>~/.ssh/config</code> file or use the alternate command line as shown in the SSH section above.

== Multiple CPU cores ==
Ways to start multi-core (from http://www.comsol.com/support/knowledgebase/1096/)
comsol -np 8

export COMSOL_NUM_THREADS=8 # automatically provided on ''Carbon''
comsol
* COMSOL does not benefit from hyperthreading (oversubscribing the physical core count).

== Batch jobs ==

* http://www.comsol.com/support/knowledgebase/1001/ .
* [[Media:HPC 2012-05-16 Comsol parallel usage.pdf]] (Modified from http://www.comsol.com/support/knowledgebase/1001/files/4027/cluster_install_linux_42a.pptx – Appendix slide, PBS)
* '''Note:''' MPD is not supported on Carbon (I patched the main comsol script), and no longer necessary at all in Comsol-4.3.

=== Sample Job file ===
See:
$COMSOL_HOME/sample.job

== Important COMSOL Support Articles ==
* Toplevel suport page: https://www.comsol.com/support

=== KB Articles ===
* [https://www.comsol.com/support/knowledgebase/1001 Running COMSOL® in parallel on clusters]
* [https://www.comsol.com/support/knowledgebase/1250 Running parametric sweeps, batch sweeps, and cluster sweeps from the command line]
* [https://www.comsol.com/support/knowledgebase/1255 Reducing the amount of solution data stored in a model]
* [https://www.comsol.com/support/knowledgebase/830 Handling large models]

=== Blog-style articles ===
* [https://www.comsol.com/blogs/the-power-of-the-batch-sweep/ The Power of the Batch Sweep]
* [https://www.comsol.com/blogs/how-to-run-simulations-in-batch-mode-from-the-command-line/ How to Run Simulations in Batch Mode from the Command Line]

HPC/Applications/comsol

2025-10-03T15:11:47Z

Stern: /* License servers */

== Introduction ==
The [http://www.comsol.com/products/multiphysics/ '''COMSOL Multiphysics'''] application suite
is available for Mac, Windows, and Linux.

== Access ==
[[Image:HPC Comsol licensed modules.png|right|450px|thumb|COMSOL features licensed at CNM as of 2024-10.]]
=== COMSOL Multiphysics ===
Access to '''COMSOL Multiphysics''', the main application with both a graphical user interface and batch computing capabilities, will be granted '''''upon request''''' .

You must be:
* An '''''Argonne employee'''''  in one of the following roles:
** '''''NST staff member'''''  (including PostDocs, joint appointments, and students at NST).
** '''''CNM Facility User'''''  (including ''non-NST Argonne employees''), with an active proposal that has COMSOL in scope. Please see [[HPC/FAQ#Access_and_proposals]].

The set of available COMSOL modules that we have licensed is shown at right.

* The native desktop version can be installed only on ''Argonne-owned'' computers (laptop or desktop). Please contact [mailto:help@anl.gov?Subject=COMSOL%20license%20access%20request CNM support]. Make sure to review [http://www.comsol.com/support/updates/ Product Updates].
* To run COMSOL on our HPC system ''Carbon'', as GUI or as batch computing jobs, your proposal must have ''Carbon'' access in scope.

=== COMSOL Server ===
''CNM Facility Users'' who are not Argonne employees are not eligible to access
the full '''COMSOL Multiphysics''' product, in compliance with its licensing terms and conditions.
However, the [https://www.comsol.com/comsol-server '''COMSOL Server'''] product offers an alternative
albeit more narrow means of access.

All users, including '''''non-Argonne users''''' may be able to run, over a web interface in their browser,
pre-parameterized "fixed-model" simulations that were made with the '''COMSOL Application Builder''' (requires Windows).

To inquire about using COMSOL server applications,
please contact us.

We will review possible purchase paths for COMSOL Server, and the ''considerable effort'' likely required by users and the CNM to develop and deploy a COMSOL application.

== Support ==
* For questions on how to run or start COMSOL:
*: Submit an [[HPC/Support | '''IT support request''']]
* For questions related to most COMSOL data or scripts, contact COMSOL support at:
*: [mailto:support@comsol.com '''support'''@comsol.com]
* '''Exception:''' If your support question might contain data that is subject to '''US Export Control''' regulations, contact instead:
*: [mailto:us-export-control@us.comsol.com '''us-export-control'''@'''us'''.comsol.com]

== The modes to run COMSOL ==
There are several ways of running COMSOL.

=== Running the desktop application ===
To edit models and analyze results, use the COMSOL graphical user interface (GUI). You can run it in several ways, each explained in more detail in the following sections.

# Run on Carbon, display on your desktop using the [http://en.wikipedia.org/wiki/X_Window_System X Window system (X11)].
#* Requires an ''X11 Server'' application running on your client machine, and SSH X11 forwarding.
#* The X11 protocol can be slow over LAN.
# Run on Carbon within a [[HPC/Network Access/Virtual Desktop | virtual desktop environment (VNC)]].
#* Requires a VNC client on your machine.
#* It can be confusing where VNC and the app runs.
#* The virtual desktop presents a (deliberatly) sparse windowing environment – Copy&Paste can be difficult to use.
# Run on your own computer, remotely checking out Carbon's license.
#* Requires a full installation of COMSOL on your desktoo or laptop computer.
#* You will require access to the COMSOL license server running on Carbon, either through VPN (fairly easy) or SSH tunneling (can be intimidating for first-time users). See below for details.

=== Running a batch calculation ===
When the computational stage of models takes longer than a few minutes to complete, it will be beneficial to submit save the model file and run it in parallel on Carbon.
See the [[#Batch jobs|section below]].

== Use with X11 ==
For casual use, and with [[HPC/Network Access/SSH Tunnel Setup on Linux and MacOS | X11 tunneling to Carbon]] set up, simply add to your <code>~/.bashrc</code> file on Carbon:
module load comsol

Then, from the command line on a Carbon login node, start the COMSOL GUI:
comsol &

License configuration is automatic on Carbon.

Comsol will use a number of cores to run calculations multithreaded by default. (On Carbon, COMSOL_NUM_THREADS is set.)

'''Quit the application''' as soon as you are done. Only one user can use Comsol at a time.

== Use with VNC ==
See general [[HPC/Network Access/Virtual Desktop | instructions on using VNC]], and the COMSOL example therein.

== Native desktop version ==

To get the fastest response from the COMSOL GUI, run a native COMSOL version on your desktop machine or laptop, called ''client machine'' below.

Enter license details either in the COMSOL Setup configuration dialog.

== Native desktop version — License configuration ==
A client's COMSOL license needs to be configured depending on how the client reaches CNM network resources. Follow the appropriate section:
* [[#License configuration when inside CNM or using VPN]]
* [[#License configuration when outside CNM and using SSH Tunneling]]

=== License configuration when inside CNM or using VPN ===
==== Ensure correct DNS search domains ====

* Inspect the client machine's network configuration.
* Ensure that '''two''' entries are listed as DNS Search Domains for the current network profile:
nst.anl.gov
cnm.anl.gov
: (The order of entries does not matter for the purposes considered here.)
* If the client machine is ''wired to CNM networks'', the domains are normally provided correctly by DHCP.
* If the client machine is ''connected over VPN'', you may need edit the respective network profile.
:* On a Mac, find the VPN configuration details under System Preferences, Network, Advanced… . You may need to unlock the Network panel first before you can make changes under "Advanced…". It appears that those changes are being saved and will be activated automatically the next time that this VPN session gets started.
: [[Image:VNC_add_cnm_to_DNS_1.png|400px]] [[Image:VNC_add_cnm_to_DNS_2.png|400px]]
:* On Windows or Linux machines, take ''analogous'' steps, using the platform's own configuration tools.

* Exclude license manager components from client desktop installation. (Those are all handled server-side.)

==== License format ====
* Floating Network License (FNL), "Three-server redundancy"

==== License servers ====
* On client machines, while running the COMSOL installer or license configuration, specify license servers and ports as follows:
1718 clicense1
1718 clicense2
1718 clicense3
# Do not use:
# clicense1<strike>.cnm.anl.gov</strike>
# clicense2<strike>.cnm.anl.gov</strike>
# clicense3<strike>.cnm.anl.gov</strike>

: '''Note:''' Use the ''short host name'' form exactly as given here. (Technical Reason: The license must be usable both on Carbon's internal network as well as on divisional networks that are outside of Carbon).

'''Advanced:''' A client machine's COMSOL license file <code><COMSOLDIR>/Multiphysics/license/license.dat</code> should look as follows:
<source lang="make">
SERVER clicense1 ANY 1718
SERVER clicense2 ANY 1718
SERVER clicense3 ANY 1718
USE_SERVER
</source>

<code><COMSOLDIR></code> is typically located at:
* '''Windows''': C:\Program files\COMSOL\COMSOL''nn''
* '''Linux''': /usr/local/comsol''nn''
* '''Mac''': /Applications/COMSOL''nn''
("''nn''" is the application's abbreviated version number.)

==== Network Troubleshooting ====
When using the license access model descibed here (as opposed to tunneled over SSH), ensure the following:
* Verify that short host names can be resolved to IP addresses. Either one of the following commands should produce sensible output:
host clicense1
: or:
ping -c 3 clicense1
* Verify that ''both license ports on the same server'' can be reached from your client machine:
telnet '''clicense1''' 1718
telnet '''clicense1''' 1719
: or:
telnet '''clicense2''' 1718
telnet '''clicense2''' 1719
: A ''successful'' connection looks like:
Trying 146.139.x.y...
Connected to clicense1.cnm.anl.gov.
Escape character is '^]'.
:: Do not enter any data. Instead, close each connection by hitting <code>Ctrl</code><code>]</code> (closing square bracket), followed by <code>Ctrl</code><code>D</code>.
: A ''failed'' connection gives one of these errors:
telnet: connect to address 146.139.x.y: Operation timed out
: or:
telnet: connect to address 146.139.x.y: Connection refused
:: In these cases, ask that your IP firewall configuration be reviewed and changed.

=== License access when outside CNM ===
If you're comfortable with ssh and tunneling, set up client licensing as follows:

==== Configure COMSOL ====
The following needs to be done just once, typically during COMSOL installation:

In the COMSOL Installer, or in COMSOL Preferences, enter:
* '''License format:''' Floating Network License (FNL), "Port number and Hostname"
:: "Three-server redundancy" cannot be used because is cannot (easily) be accessed over SSH. Configure with just one server, and in rare tweak SSH to tunnel to the other server.
* '''Server:''' <code>127.0.0.1</code>
:: Do not use "localhost", as COMSOL interprets this incorrectly to mean that your machine ''is'' a license server.

* '''Port:''' <code>1718</code>


==== Provide server connection ====
This step needs to be done prior to running COMSOL in your desktop session:
* Connect to <code>mega</code> with the following ssh command:
ssh -L 1718:'''clicense1''':1718 -L 1719:'''clicense1''':1719 mega.cnm.anl.gov
* Leave the command running between invocations of COMSOL.

==== Run COMSOL ====
* Ensure your connection to "mega" is still running. If not, restart it.
* Start the COMSOL application.

** If you get an error that the number of licenses is "exhausted", that is confirmation that you correctly reached the license server, though other users are using licenses.
** If the license access attempt takes a long time or hangs, the service might temporarily run on a secondary server. In this case, stop the mega session and restart it, connecting to the alternate license server:
ssh -L 1718:clicense'''2''':1718 -L 1719:clicense'''2''':1719 mega.cnm.anl.gov

== License Troubleshooting ==
The following are typical error messages.
=== Could not obtain license for COMSOL Multiphysics GUI ===
* Error details:
License error: -4.
'''Licensed number of users already reached.'''
Feature: COMSOLGUI
License path: /Users/stern/Applications/COMSOL52a/license/license.dat:
FlexNet Licensing error:-4,132
* Reason: The connection to the license server succeeded, but the license token(s) you requested are in use.
* Solution: Wait a bit, or [http://carbonmgmt/ganglia/lic.php identify who's using COMSOL] and politely ask the user when they might be finished with their work.

=== License server machine is down or not responding ===
* Error details:
License error: -96.
See the system administrator about starting the license server system, or
make sure you're referring to the right host (see LM_LICENSE_FILE).
Feature: SERIAL
Hostname: clicense3
License path: /Users/stern/Applications/COMSOL52a/license/license.dat:
FlexNet Licensing error:-96,7
* Reason: The license servers cannot be reached at all. The <code>ping</code> or <code>host</code> tests mentioned above fail.
* Solution: Review the client license file and DNS lookup domains.

=== License file does not support this version ===
* Error details:
Feature: SERIAL
'''Application version > License version: 5.21 > 3.5'''
License path: /Users/stern/Applications/COMSOL52a/license/license.dat:
FlexNet Licensing error:-21,126
* Reason: Only the first of two steps for connecting to the license servers succeeds, which may happen when the redundant license service runs on the alternate server instead of the one you're connected to.
* Solution: When using SSH, try <code>clicense2</code> instead of <code>clicense1</code>, or vice versa. Revisit your <code>~/.ssh/config</code> file or use the alternate command line as shown in the SSH section above.

== Multiple CPU cores ==
Ways to start multi-core (from http://www.comsol.com/support/knowledgebase/1096/)
comsol -np 8

export COMSOL_NUM_THREADS=8 # automatically provided on ''Carbon''
comsol
* COMSOL does not benefit from hyperthreading (oversubscribing the physical core count).

== Batch jobs ==

* http://www.comsol.com/support/knowledgebase/1001/ .
* [[Media:HPC 2012-05-16 Comsol parallel usage.pdf]] (Modified from http://www.comsol.com/support/knowledgebase/1001/files/4027/cluster_install_linux_42a.pptx – Appendix slide, PBS)
* '''Note:''' MPD is not supported on Carbon (I patched the main comsol script), and no longer necessary at all in Comsol-4.3.

=== Sample Job file ===
See:
$COMSOL_HOME/sample.job

== Important COMSOL Support Articles ==
* Toplevel suport page: https://www.comsol.com/support

=== KB Articles ===
* [https://www.comsol.com/support/knowledgebase/1001 Running COMSOL® in parallel on clusters]
* [https://www.comsol.com/support/knowledgebase/1250 Running parametric sweeps, batch sweeps, and cluster sweeps from the command line]
* [https://www.comsol.com/support/knowledgebase/1255 Reducing the amount of solution data stored in a model]
* [https://www.comsol.com/support/knowledgebase/830 Handling large models]

=== Blog-style articles ===
* [https://www.comsol.com/blogs/the-power-of-the-batch-sweep/ The Power of the Batch Sweep]
* [https://www.comsol.com/blogs/how-to-run-simulations-in-batch-mode-from-the-command-line/ How to Run Simulations in Batch Mode from the Command Line]

HPC/Network Access/SSH Tunnel Setup and Use

2025-09-23T18:05:10Z

Stern: /* Config file contents */

The following sections document how to establish ssh tunneling on machines running [http://en.wikipedia.org/wiki/OpenSSH OpenSSH] or close relatives.
OpenSSH is available on all major operating systems: Linux, macOS, Windows, and on mobile.


== Manual setup ==
[[Image:HPC 2012-02-06 screenshot sketch ssh tunneling windows.png|350px|right]]
'''On your computer''', run the following commands:

0. (Windows only) Start your X11 server if you have one.

1. Open a terminal window (such as Mac Terminal, Cygwin, <strike>PuTTY</strike>, or xterm).

2. Type:
ssh -v -L 33301:carbon:22 ''username''@mega.cnm.anl.gov
: '''Do not type any further commands in this window.'''

3. Minimize that window - no further commands need be typed there.

4. Open another terminal window.

5. Type:
ssh -Y -p 33301 ''username''@localhost
:* Replace <code>''username''</code> by the name of your Argonne account name.
:* Steps 4 and 5 can be repeated as often as desired.

Do not type any commands at a prompt on mega.

== Using an ssh config file ==
To simplify making connections from your computer, create or use an ssh configuration file on it.

=== Config file contents ===
'''On your computer''':
* Close any Carbon windows opened previously.
* Open a new terminal window.
** Linux: xterm
** Mac: Terminal.app
** Windows: a Cygwin window.
* Copy&paste the following text into your <code>~/.ssh/config</code> file.

NoHostAuthenticationForLocalhost yes

Host mega
Hostname mega.cnm.anl.gov
User ''argonne_user_name''
LocalForward 33301 carbon:22
LocalForward 33380 carbonmgmt:80

Host carbon
Hostname localhost
User ''argonne_user_name''
Port 33301
ForwardX11 yes
ForwardX11Trusted yes
AddKeysToAgent yes
'''MACs hmac-sha2-512,hmac-sha2-256'''
'''RequiredRSASize 1024'''

* '''Replace''' the string ''argonne_user_name'' with the name of your Argonne domain account.
* The <code>MACs</code> line is required on Windows 11 Subsystem for Linux (WSL).
* Remove write permissions to the config file, its parent directory, and your home for anyone but you:
<syntaxhighlight lang="bash">
chmod go-rwx ~/.ssh ~/.ssh/config
chmod go-w ~
</syntaxhighlight>
:* '''Note for Cygwin users:''' To paste text into a Cygwin window, right-click with your mouse or use the Shift-Ins keyboard shortcut.

:* Simply Copy & Paste the line from this browser window to your terminal.
* Advanced users: See the ssh_config man page for syntax and options of the config file.

=== Connecting with a config file ===
With the config file in place, the commands in section [[#Manual setup]] above simplify to:

: 1. In one window, run the following command, then minimize the window:
ssh -v mega
: Do not type any further commands here. (You can use the -N option to intentionally suppress a command prompt.)

: 2. Use other terminals as often as needed to log in interactively:
ssh carbon

=== Advanced: Connecting to a ''specific'' login node ===
The configuration described above uses a generic host name {{host|carbon}}, which at present will lead to one specific login node.
However, you may want to connect to another node that you have used previously, such as to reconnect to a VNC sesssion.
To do so, change your .ssh/config to forward additional ports and set up more specific <code>Host</code> aliases.

==== Modify configuration file ====
* Similar to the above, copy&paste the following text into your <code>~/.ssh/config</code> file:
NoHostAuthenticationForLocalhost yes

Host mega
Hostname mega.cnm.anl.gov
User ''argonne_user_name''
LocalForward 33301 carbon:22
LocalForward 33305 clogin5:22
LocalForward 33306 clogin6:22
LocalForward 33380 carbonmgmt:80

Host carbon
Port 33301
Host clogin5
Port 33305
Host clogin6
Port 33306
Host carbon clogin*
…

You can then connect to either {{host|carbon}}, or specific host names {{host|clogin5}} etc., as follows:
: 1. Run '''once''' in your desktop session:
ssh -v mega
: 2. Run '''as often as needed''':
ssh clogin
ssh clogin5
ssh clogin6
scp … # (to or from {{host|clogin}})
…

== Test remote graphics using X11 ==
# In a terminal '''connected to a Carbon login node''', run the command:
#:: <source lang="bash">echo $DISPLAY</source>
#:: You should get something like:
#:: <code>localhost:''17.0''</code>
#: The actual number will vary. [[Image:HPC 2011-06-23 xload example.png|right|60px]]
# Start a simple X11 program:
#:: <source lang="bash">xload</source>
#: You should see a small window pop up on your screen, labeled "login5" or "login6".
# Close the window using its close button or press Ctrl-C in the terminal window.
# Now use graphical applications on Carbon, for instance: [[Image:HPC 2011-06-23 gnuplot example.png|right|300px]]
#:: <source lang="bash">gnuplot</source>
#: This will start the gnuplot interpreter in the terminal.
#* Create a sample plot. At the <code>gnuplot> </code> prompt, type:
#*: <source lang="bash">plot sin(x)</source>
#* Exit the program:
#*: <source lang="bash">exit</source>

== File transfer ==
* There are many ways to transfer files through an ssh tunnel. However, you must always initiate the connection from the remote machine (your home computer), and either ''push'' files to Carbon or ''pull'' files from Carbon.
* With the above tunnel setup you cannot initiate a file transfer from Carbon or mega.
* Do not use mega as an intermediate. Files stored there wil be deleted.

; Prerequisite: The command examples below require that a tunnel-establishing '''''connection to <code>mega</code> is open'''''. Follow section [[#Manual setup]] or [[#Connecting with a config file]] above to open or re-open this connection if needed.

=== Without SSH config file ===


Run the following '''example commands on your machine'''.

; Copy ''to'' Carbon – "push":
scp -P 33301 -p 'foofile' ''argonne_user_name''@localhost:'dir/on/carbon/'

; Copy ''from'' Carbon – "pull":
scp -P -p ''argonne_user_name''@localhost:'dir/on/carbon/name_on_carbon' destination_on_your_machine/

* Type <code>localhost</code> as written. Together with the option <code>-P 33301</code> (or <code>-o port=''number''</code> for older scp versions) it refers to the tunnel that you have set up earlier.
* Replace ''<code>argonne_user_name</code>'' by the user name you have at Argonne.

=== With an SSH config file ===
Run the following '''example commands on your machine'''.

==== Copy a file ''to'' Carbon (''push'' from outside) ====
* With the above ssh/.config file in place:
scp -p 'foofile' carbon:'dir/on/carbon/'

==== Copy a whole directory ''to'' Carbon ====
Short of using tar or zip:
scp -rp 'local/path/' carbon:'dir/on/carbon/'
This method does not preserve symbolic links.

==== Copy a file ''from'' Carbon (''pull'' from outside) ====
scp -p carbon:remote/path/foo local/path/
scp -p carbon:remote/path/foo .
Note the trailing "." in the last example to copy to your current directory.
; Reverse tunneling
The file copy operations above must all be initiated on the outside host.
It is possible to set up tunnels to be able to initiate file operations on the inside.
However, this is more complex and will be left as an exercise for advanced users.
It involves setting up:
# agent forwarding
# a <code>.ssh/config</code> file on Carbon, and
# specifying <code>-R</code> command line options or <code>RemoteForward</code> config file directives when accessing {{host|carbon}} at the second stage.

==== Bonus – Intranet web access ====
Open the following links in a browser ''on your external workstation'':
https://localhost:33343/cnm/HPC
http://localhost:33380/

==== Copy a file ''to'' Carbon from a ''third host'' ====
0. Similar to the procedure in section [[#Manual setup]] above, open two interactive sessions to the remote computer.

1. From the first session, open a connection to Mega and forward a port:
ssh -v -L 33301:carbon:22 ''username''@mega.cnm.anl.gov
:* '''Do not type any further commands there.'''

2. In the second session, test if you can reach Carbon interactively.
ssh -Y -p 33301 -o NoHostAuthenticationForLocalhost=yes username@localhost

3. Close the Carbon connection from step 2.
exit
: You should arrive back at the prompt of the third host.

4. From the second session again, initiate the file transfer as:
scp -v -p -o port=33301 -o NoHostAuthenticationForLocalhost=yes \
''filename_remote'' ''username''@localhost''':'''[''target_directory/''][''filename_on_carbon'']
:* Replace <code>''username''</code> by the name of your Argonne account name.
:* Note the ":" at the end of the host name, which tells the <code>scp</code> command that the copy target is on a remote host.
:* The file named <code>''filename_remote''</code> will arrive in your Carbon home directory under this name, unless you specify a <code>''target_directory''</code> and/or a <code>''filename_on_carbon''</code>.

5. When done, press Ctrl-C in the first session to terminate the connection to mega.

== Advanced: Speeding up logins using ssh keys ==

Our SSH gatway host itself does not accept SSH keys – you must always type your password for the initial connection to Mega.

Once you have established this initial connection and your tunnels are configured, you can use SSH keys to log in to Carbon login nodes without further passwords.

=== Create a key ===
* Create an ssh key pair key ''if you do not already have one'' :
yourcomputer$ '''ssh-keygen -t rsa'''

The output will look like:
Generating public/private rsa key pair.
Enter file in which to save the key (/Users/home_id/.ssh/id_rsa):
Enter passphrase (empty for no passphrase): <-- Choose and enter a password.
Enter same passphrase again:

Your identification has been saved in /Users/home_id/.ssh/id_rsa.
Your public key has been saved in /Users/home_id/.ssh/id_rsa.pub.
The key fingerprint is:
42:c3:72:fb:4c:be:c6:80:81:bc:8b:73:d8:88:84:2e home_id@yourcomputer.local

* Inspect the .ssh directory '''on your computer'''. It should contain the following files:
$ ls -1 ~/.ssh
config
'''id_rsa'''
'''id_rsa.pub'''
known_hosts
random_seed
: id_* are the just created key files.

=== Using an ssh-agent ===
On modern systems, the mere presence of private keys with their default names triggers handling through ''ssh-agent'' and brings up a passphrase dialog at the start of the desktop session or when a key is first requested for an outgoing connection. This is the case on MacOS X since 10.5 (Leopard), where key management is secure and largely automated through its general ''keychain'' concept.

On older systems, you need to (a) set up the session to run under ''ssh agent'' and (b) request the passphrase upon login. Circumstances vary widely. Ask [http://www.google.com/search?hl=en&q=ssh-agent Google] or your local admin for help. See also the page [http://kimmo.suominen.com/docs/ssh/ ''Getting started with SSH''].

=== Place your public key on Carbon ===
[[Image:HPC 2012-02-06 screenshot ssh agent password request.png|350px|thumb|Password dialog on connecting to {{host|carbon}} once keys are in place.]]
To productively use your ssh key pair, you need to transfer the public half to Carbon.

Requirements:
* you have a public/private ssh key pair on your machine (called '''yourcomputer''' in this example).
* the public file on '''yourcomputer''' is named <code>~/.ssh/id_rsa.pub</code> or <code>~/.ssh/id_dsa.pub</code> (either one will work).
* the ssh host alias <code>mega</code> has been set up on '''yourcomputer''' in <code>~/.ssh/config</code> and defines tunnels using <code>LocalForward</code> declarations.
* the tunnel connection for '''mega''' is open, i.e., on '''yourcomputer''', <code>ssh mega</code> runs in another terminal.
* the ssh alias {{host|carbon}} has been set up as described above.
* an interactive session <code>ssh carbon</code> succeeds.

 
Then, to transfer your public key from '''yourcomputer''' to the Carbon login nodes, type:
cat ~/.ssh/id_rsa.pub | ssh carbon "umask 033; mkdir -p .ssh; cat >> .ssh/authorized_keys"
This command will append the contents of your ssh public key on '''yourcomputer''' to an <code>authorized_keys</code> file on Carbon. The command above is more reliable than cut&paste across interactive terminal sessions. If you created a dsa-type key, or chose your own name for the key pair, you'll have to cat the appropriate file instead. Remember to use the ''public'' part of the key, i.e. the file <code>id_foo.pub</code>.

== Troubleshooting ==

=== General connection issues ===
Run <code>ssh</code> with one or more <code>-v</code> options ("verbosity"), and record the output for diagnostic (copy&paste into a file). E.g.:
<source lang="bash">
ssh -v -v [other options and arguments …]
</source>
If you cannot resolve the issue yourself, follow [[HPC/Support]].

=== Public keys ===
If public keys do not work, use your password to gain access, then check the following:
; Permissions: The directories and the <code>authorized_keys</code> file must, at the least, not be writable by anyone except yourself; other permissions are recommended to be restrictive as well:
login5$ ls -ld ~ ~/.ssh ~/.ssh/authorized_keys
drwxr-xr-x 28 argonne_id users 4096 Feb 22 13:52 /home/argonne_id
drwx------ 4 argonne_id argonne_id 4096 Feb 18 21:25 /home/argonne_id/.ssh
-rw------- 1 argonne_id argonne_id 829 Oct 9 10:55 /home/argonne_id/.ssh/authorized_keys
: To change:
login5$ '''chmod go-w ~'''
login5$ '''chmod go-rwx ~/.ssh ~/.ssh/authorized_keys'''
; Corruption of <code>authorized_keys</code> file: The file should contain public keys in OpenSSH format, ''one per line'', without additional line breaks. One way to check is:
yourcomputer$ cut -c1-50 ~/.ssh/authorized_keys
ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAIEjdsfjJSAHkjh88y
ssh-dss AAAAB3NzaC1kc3MAAACBAkkjsadfJFg7EFjDJhWEjk
The output should ''look similar'' to the sample above. Use your favorite text editor to join erroneously broken lines.
Note that there should '''not''' be anything like <code>BEGIN RSA PRIVATE KEY</code>, which would obviously indicate a private, rather than public key.
; Verify that an ssh-agent is running:
To verify that an ssh-agent is running and visible to your shell, check for the presence of the environment variable <code>SSH_AUTH_SOCK</code>. To list the keys it has loaded, use <code>ssh-add -l</code> (lowercase "L"). Typical output is as follows:
yourcomputer$ '''echo $SSH_AUTH_SOCK'''
/tmp/ssh-HCvZKt5478/agent.5478
yourcomputer$ '''ssh-add -l'''
1024 b9:65:98:c0:7c:25:da:b6:86:48:98:6b:2a:5a:9d:21 /Users/home_id/.ssh/id_rsa (RSA)

[[Category:HPC|Network]]

HPC/Network Access/SSH Tunnel Setup and Use

2025-09-23T18:04:29Z

Stern: /* Using an ssh config file */

The following sections document how to establish ssh tunneling on machines running [http://en.wikipedia.org/wiki/OpenSSH OpenSSH] or close relatives.
OpenSSH is available on all major operating systems: Linux, macOS, Windows, and on mobile.


== Manual setup ==
[[Image:HPC 2012-02-06 screenshot sketch ssh tunneling windows.png|350px|right]]
'''On your computer''', run the following commands:

0. (Windows only) Start your X11 server if you have one.

1. Open a terminal window (such as Mac Terminal, Cygwin, <strike>PuTTY</strike>, or xterm).

2. Type:
ssh -v -L 33301:carbon:22 ''username''@mega.cnm.anl.gov
: '''Do not type any further commands in this window.'''

3. Minimize that window - no further commands need be typed there.

4. Open another terminal window.

5. Type:
ssh -Y -p 33301 ''username''@localhost
:* Replace <code>''username''</code> by the name of your Argonne account name.
:* Steps 4 and 5 can be repeated as often as desired.

Do not type any commands at a prompt on mega.

== Using an ssh config file ==
To simplify making connections from your computer, create or use an ssh configuration file on it.

=== Config file contents ===
'''On your computer''':
* Close any Carbon windows opened previously.
* Open a new terminal window.
** Linux: xterm
** Mac: Terminal.app
** Windows: a Cygwin window.
* Copy&paste the following text into your ~/.ssh/config file.

NoHostAuthenticationForLocalhost yes

Host mega
Hostname mega.cnm.anl.gov
User ''argonne_user_name''
LocalForward 33301 carbon:22
LocalForward 33380 carbonmgmt:80

Host carbon
Hostname localhost
User ''argonne_user_name''
Port 33301
ForwardX11 yes
ForwardX11Trusted yes
AddKeysToAgent yes
'''MACs hmac-sha2-512,hmac-sha2-256'''
'''RequiredRSASize 1024'''

* '''Replace''' the string ''argonne_user_name'' with the name of your Argonne domain account.
* The <code>MACs</code> line is required on Windows 11 Subsystem for Linux (WSL).
* Remove write permissions to the config file, its parent directory, and your home for anyone but you:
<syntaxhighlight lang="bash">
chmod go-rwx ~/.ssh ~/.ssh/config
chmod go-w ~
</syntaxhighlight>
:* '''Note for Cygwin users:''' To paste text into a Cygwin window, right-click with your mouse or use the Shift-Ins keyboard shortcut.

:* Simply Copy & Paste the line from this browser window to your terminal.
* Advanced users: See the ssh_config man page for syntax and options of the config file.

=== Connecting with a config file ===
With the config file in place, the commands in section [[#Manual setup]] above simplify to:

: 1. In one window, run the following command, then minimize the window:
ssh -v mega
: Do not type any further commands here. (You can use the -N option to intentionally suppress a command prompt.)

: 2. Use other terminals as often as needed to log in interactively:
ssh carbon

=== Advanced: Connecting to a ''specific'' login node ===
The configuration described above uses a generic host name {{host|carbon}}, which at present will lead to one specific login node.
However, you may want to connect to another node that you have used previously, such as to reconnect to a VNC sesssion.
To do so, change your .ssh/config to forward additional ports and set up more specific <code>Host</code> aliases.

==== Modify configuration file ====
* Similar to the above, copy&paste the following text into your <code>~/.ssh/config</code> file:
NoHostAuthenticationForLocalhost yes

Host mega
Hostname mega.cnm.anl.gov
User ''argonne_user_name''
LocalForward 33301 carbon:22
LocalForward 33305 clogin5:22
LocalForward 33306 clogin6:22
LocalForward 33380 carbonmgmt:80

Host carbon
Port 33301
Host clogin5
Port 33305
Host clogin6
Port 33306
Host carbon clogin*
…

You can then connect to either {{host|carbon}}, or specific host names {{host|clogin5}} etc., as follows:
: 1. Run '''once''' in your desktop session:
ssh -v mega
: 2. Run '''as often as needed''':
ssh clogin
ssh clogin5
ssh clogin6
scp … # (to or from {{host|clogin}})
…

== Test remote graphics using X11 ==
# In a terminal '''connected to a Carbon login node''', run the command:
#:: <source lang="bash">echo $DISPLAY</source>
#:: You should get something like:
#:: <code>localhost:''17.0''</code>
#: The actual number will vary. [[Image:HPC 2011-06-23 xload example.png|right|60px]]
# Start a simple X11 program:
#:: <source lang="bash">xload</source>
#: You should see a small window pop up on your screen, labeled "login5" or "login6".
# Close the window using its close button or press Ctrl-C in the terminal window.
# Now use graphical applications on Carbon, for instance: [[Image:HPC 2011-06-23 gnuplot example.png|right|300px]]
#:: <source lang="bash">gnuplot</source>
#: This will start the gnuplot interpreter in the terminal.
#* Create a sample plot. At the <code>gnuplot> </code> prompt, type:
#*: <source lang="bash">plot sin(x)</source>
#* Exit the program:
#*: <source lang="bash">exit</source>

== File transfer ==
* There are many ways to transfer files through an ssh tunnel. However, you must always initiate the connection from the remote machine (your home computer), and either ''push'' files to Carbon or ''pull'' files from Carbon.
* With the above tunnel setup you cannot initiate a file transfer from Carbon or mega.
* Do not use mega as an intermediate. Files stored there wil be deleted.

; Prerequisite: The command examples below require that a tunnel-establishing '''''connection to <code>mega</code> is open'''''. Follow section [[#Manual setup]] or [[#Connecting with a config file]] above to open or re-open this connection if needed.

=== Without SSH config file ===


Run the following '''example commands on your machine'''.

; Copy ''to'' Carbon – "push":
scp -P 33301 -p 'foofile' ''argonne_user_name''@localhost:'dir/on/carbon/'

; Copy ''from'' Carbon – "pull":
scp -P -p ''argonne_user_name''@localhost:'dir/on/carbon/name_on_carbon' destination_on_your_machine/

* Type <code>localhost</code> as written. Together with the option <code>-P 33301</code> (or <code>-o port=''number''</code> for older scp versions) it refers to the tunnel that you have set up earlier.
* Replace ''<code>argonne_user_name</code>'' by the user name you have at Argonne.

=== With an SSH config file ===
Run the following '''example commands on your machine'''.

==== Copy a file ''to'' Carbon (''push'' from outside) ====
* With the above ssh/.config file in place:
scp -p 'foofile' carbon:'dir/on/carbon/'

==== Copy a whole directory ''to'' Carbon ====
Short of using tar or zip:
scp -rp 'local/path/' carbon:'dir/on/carbon/'
This method does not preserve symbolic links.

==== Copy a file ''from'' Carbon (''pull'' from outside) ====
scp -p carbon:remote/path/foo local/path/
scp -p carbon:remote/path/foo .
Note the trailing "." in the last example to copy to your current directory.
; Reverse tunneling
The file copy operations above must all be initiated on the outside host.
It is possible to set up tunnels to be able to initiate file operations on the inside.
However, this is more complex and will be left as an exercise for advanced users.
It involves setting up:
# agent forwarding
# a <code>.ssh/config</code> file on Carbon, and
# specifying <code>-R</code> command line options or <code>RemoteForward</code> config file directives when accessing {{host|carbon}} at the second stage.

==== Bonus – Intranet web access ====
Open the following links in a browser ''on your external workstation'':
https://localhost:33343/cnm/HPC
http://localhost:33380/

==== Copy a file ''to'' Carbon from a ''third host'' ====
0. Similar to the procedure in section [[#Manual setup]] above, open two interactive sessions to the remote computer.

1. From the first session, open a connection to Mega and forward a port:
ssh -v -L 33301:carbon:22 ''username''@mega.cnm.anl.gov
:* '''Do not type any further commands there.'''

2. In the second session, test if you can reach Carbon interactively.
ssh -Y -p 33301 -o NoHostAuthenticationForLocalhost=yes username@localhost

3. Close the Carbon connection from step 2.
exit
: You should arrive back at the prompt of the third host.

4. From the second session again, initiate the file transfer as:
scp -v -p -o port=33301 -o NoHostAuthenticationForLocalhost=yes \
''filename_remote'' ''username''@localhost''':'''[''target_directory/''][''filename_on_carbon'']
:* Replace <code>''username''</code> by the name of your Argonne account name.
:* Note the ":" at the end of the host name, which tells the <code>scp</code> command that the copy target is on a remote host.
:* The file named <code>''filename_remote''</code> will arrive in your Carbon home directory under this name, unless you specify a <code>''target_directory''</code> and/or a <code>''filename_on_carbon''</code>.

5. When done, press Ctrl-C in the first session to terminate the connection to mega.

== Advanced: Speeding up logins using ssh keys ==

Our SSH gatway host itself does not accept SSH keys – you must always type your password for the initial connection to Mega.

Once you have established this initial connection and your tunnels are configured, you can use SSH keys to log in to Carbon login nodes without further passwords.

=== Create a key ===
* Create an ssh key pair key ''if you do not already have one'' :
yourcomputer$ '''ssh-keygen -t rsa'''

The output will look like:
Generating public/private rsa key pair.
Enter file in which to save the key (/Users/home_id/.ssh/id_rsa):
Enter passphrase (empty for no passphrase): <-- Choose and enter a password.
Enter same passphrase again:

Your identification has been saved in /Users/home_id/.ssh/id_rsa.
Your public key has been saved in /Users/home_id/.ssh/id_rsa.pub.
The key fingerprint is:
42:c3:72:fb:4c:be:c6:80:81:bc:8b:73:d8:88:84:2e home_id@yourcomputer.local

* Inspect the .ssh directory '''on your computer'''. It should contain the following files:
$ ls -1 ~/.ssh
config
'''id_rsa'''
'''id_rsa.pub'''
known_hosts
random_seed
: id_* are the just created key files.

=== Using an ssh-agent ===
On modern systems, the mere presence of private keys with their default names triggers handling through ''ssh-agent'' and brings up a passphrase dialog at the start of the desktop session or when a key is first requested for an outgoing connection. This is the case on MacOS X since 10.5 (Leopard), where key management is secure and largely automated through its general ''keychain'' concept.

On older systems, you need to (a) set up the session to run under ''ssh agent'' and (b) request the passphrase upon login. Circumstances vary widely. Ask [http://www.google.com/search?hl=en&q=ssh-agent Google] or your local admin for help. See also the page [http://kimmo.suominen.com/docs/ssh/ ''Getting started with SSH''].

=== Place your public key on Carbon ===
[[Image:HPC 2012-02-06 screenshot ssh agent password request.png|350px|thumb|Password dialog on connecting to {{host|carbon}} once keys are in place.]]
To productively use your ssh key pair, you need to transfer the public half to Carbon.

Requirements:
* you have a public/private ssh key pair on your machine (called '''yourcomputer''' in this example).
* the public file on '''yourcomputer''' is named <code>~/.ssh/id_rsa.pub</code> or <code>~/.ssh/id_dsa.pub</code> (either one will work).
* the ssh host alias <code>mega</code> has been set up on '''yourcomputer''' in <code>~/.ssh/config</code> and defines tunnels using <code>LocalForward</code> declarations.
* the tunnel connection for '''mega''' is open, i.e., on '''yourcomputer''', <code>ssh mega</code> runs in another terminal.
* the ssh alias {{host|carbon}} has been set up as described above.
* an interactive session <code>ssh carbon</code> succeeds.

 
Then, to transfer your public key from '''yourcomputer''' to the Carbon login nodes, type:
cat ~/.ssh/id_rsa.pub | ssh carbon "umask 033; mkdir -p .ssh; cat >> .ssh/authorized_keys"
This command will append the contents of your ssh public key on '''yourcomputer''' to an <code>authorized_keys</code> file on Carbon. The command above is more reliable than cut&paste across interactive terminal sessions. If you created a dsa-type key, or chose your own name for the key pair, you'll have to cat the appropriate file instead. Remember to use the ''public'' part of the key, i.e. the file <code>id_foo.pub</code>.

== Troubleshooting ==

=== General connection issues ===
Run <code>ssh</code> with one or more <code>-v</code> options ("verbosity"), and record the output for diagnostic (copy&paste into a file). E.g.:
<source lang="bash">
ssh -v -v [other options and arguments …]
</source>
If you cannot resolve the issue yourself, follow [[HPC/Support]].

=== Public keys ===
If public keys do not work, use your password to gain access, then check the following:
; Permissions: The directories and the <code>authorized_keys</code> file must, at the least, not be writable by anyone except yourself; other permissions are recommended to be restrictive as well:
login5$ ls -ld ~ ~/.ssh ~/.ssh/authorized_keys
drwxr-xr-x 28 argonne_id users 4096 Feb 22 13:52 /home/argonne_id
drwx------ 4 argonne_id argonne_id 4096 Feb 18 21:25 /home/argonne_id/.ssh
-rw------- 1 argonne_id argonne_id 829 Oct 9 10:55 /home/argonne_id/.ssh/authorized_keys
: To change:
login5$ '''chmod go-w ~'''
login5$ '''chmod go-rwx ~/.ssh ~/.ssh/authorized_keys'''
; Corruption of <code>authorized_keys</code> file: The file should contain public keys in OpenSSH format, ''one per line'', without additional line breaks. One way to check is:
yourcomputer$ cut -c1-50 ~/.ssh/authorized_keys
ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAIEjdsfjJSAHkjh88y
ssh-dss AAAAB3NzaC1kc3MAAACBAkkjsadfJFg7EFjDJhWEjk
The output should ''look similar'' to the sample above. Use your favorite text editor to join erroneously broken lines.
Note that there should '''not''' be anything like <code>BEGIN RSA PRIVATE KEY</code>, which would obviously indicate a private, rather than public key.
; Verify that an ssh-agent is running:
To verify that an ssh-agent is running and visible to your shell, check for the presence of the environment variable <code>SSH_AUTH_SOCK</code>. To list the keys it has loaded, use <code>ssh-add -l</code> (lowercase "L"). Typical output is as follows:
yourcomputer$ '''echo $SSH_AUTH_SOCK'''
/tmp/ssh-HCvZKt5478/agent.5478
yourcomputer$ '''ssh-add -l'''
1024 b9:65:98:c0:7c:25:da:b6:86:48:98:6b:2a:5a:9d:21 /Users/home_id/.ssh/id_rsa (RSA)

[[Category:HPC|Network]]

HPC/FAQ

2025-09-19T17:14:49Z

Stern: /* Password lockouts */

== Access and proposals ==
The HPC ''Carbon'' cluster and its associated software licenses may be accessed,
like any tool at the CNM, by one or both of the following routes:
* CNM Facility Users are eligible under [https://www.anl.gov/cnm/become-a-cnm-user '''CNM's User Access Program''']. Follow the link to learn how to initiate access under this route. Begin by registering as Facility User, then submit a proposal.
* Affiliates of [http://www.anl.gov/cnm Argonne's NST division] are eligible for discretionary access. Affiliate here refers to regular employees, Argonne/University joint appointees, visitor appointees, and student group members, being supervised by CNM staff.

The following is a summary of the main points as they pertain to ''Carbon''.

=== Legal User Agreement ===
The institution where you work or study must have a legal signed
[https://www.anl.gov/cnm/prepare-for-your-arrival '''User Facility Agreement''']
with Argonne in place before any work under an eventual proposal may be performed.

[https://aps.anl.gov/Users-Information/Legal-Financial/Argonne-User-Facility-Agreements '''Look up existing User Facility Agreements'''] and locate your institution if you have never worked with Argonne before, or you recently changed your affiliation, or are going to change it soon.

If your institution is not shown, ''get started'' with establishing a user agreement as soon as possible. Legal wheels turn slowly and your scientific work could be delayed.

=== User Registration ===
Under CNM's User Access Program, your starting point is to [https://www.anl.gov/cnm/how-to-register-to-become-a-cnm-user '''become a CNM user'''].

=== Proposal Lifecycle Overview ===

# To '''submit a proposal''', follow the instructions at the CNM's [https://www.anl.gov/cnm/cnm-user-information '''Call for Proposals'''].
# Upon successful review of a proposal and grant of an allocation, '''the contacting author''' will receive an email with subject '''User Proposal Status Notification'''.
# '''The contacting author must submit,''' for each proposal, a Safety&Data Form as instructed in the status email. This will confirm the names and badge numbers of participating users, as well as various characteristics of the proposal that we need for statistical reporting to our funding agencies.
# Participating '''users register and respond''' at various steps to have their Argonne user accounts created, activated or reactivated as needed, and have them authorized to access Carbon.
# Participating users '''perform the work proposed'''.
# [https://www.anl.gov/cnm/cnm-acknowledgment-statements-for-publications '''Credit CNM in publications'''] that result from your proposal work with us.
# After the proposal expires, our User Office will ask you to please submit:
#* An approximately 1-page '''Activity Report''' summarizing work performed and perhaps already published under the proposal. We use these reports to document the use of our facilities to funding agencies.
#* An end-of-proposal '''Satisfaction Survey''', to invite your feedback on our process.

== Account Types ==
You access our computer systems by two different user accounts:

=== User Registration Account ===
: – Looks like: <code>123456</code> – 
Your ''User Registration Account:''
* … is created when you register as CNM facility user, be it as participant or as Principal Investigator (PI), at the [https://beam.aps.anl.gov/pls/apsweb/ufr_main_pkg.usr_start_page '''User Registration'''] for the Advanced Photon Source (APS) and CNM.
* … has an all-numeric user name (no letters, no leading zeroes), which never changes – this is in fact your ''Argonne Badge Number'',
* … has its own password and [https://beam.aps.anl.gov/pls/apsweb/forgot_password.start_process '''password reset process at APS'''],

Use your User Registration Account to:
* … [https://beam.aps.anl.gov/pls/apsweb/ufr_main_pkg.usr_start_page '''update your user registration'''], such as when you change your affiliation, or to update visa information if Argonne needs them for you,
* … [https://pico.cnm.anl.gov/proposals/people/login '''submit a user proposal to CNM''']
* … [https://beam.aps.anl.gov/pls/apsweb/rt0004.intro_process '''take remote training'''] prior to beginning work, and prior to being assigned an Argonne domain account, which is described next.

=== Argonne Domain Account ===
: – Looks like: <code>'''b'''''123456''</code> or <code>''jsmith''</code> – 

Your ''Argonne Domain Account:''
* … is created '''when you first need it''', such as when you are a PI and have received the go-ahead on your first proposal with us,
* … has, at first, a user name beginning with <code>'''b'''</code>, followed by your badge number (without any leading zeroes),
* … will get, following your request, a personalized user name, such as <code>''jsmith''</code>
* … has its own password and uses its own [https://mypassword.anl.gov/ '''password reset and recovery''']. 

Use your Argonne Domain Account to:
* … submit the '''Safety&Data Form (UWS)''' of each proposal for which you are the PI or delegated spokesperson,
* … perform work – always remotely – at [[HPC|CNM's '''High Performance Computing system ''Carbon''''']], 
* … handle almost all other tasks at Argonne.

For assistance, please contact our [mailto:cnm_useroffice@anl.gov?subject=Account%20Question CNM User Office]. – Specify ''which task'' you have difficulties with.

== Getting started on a proposal ==

To start working on a proposal once it has been granted, '''action is required for each proposal''' by the proposal spokesperson (usually the Principal Investigator,PI), regardless of any previous proposals with us.
At this stage, we collect and update information about each proposal for reporting purposes to our funding agencies, and ensure that safety and training requirements are met by all participating users.

=== To get started as spokesperson or delegate ===
# Locate the email to you with subject ''User Proposal Status Notification'', sent by the CNM User Office.
#* If you are delayed in starting your proposal, you may need to expand your mailbox search time frame, to begin the search at possibly several months before present.
#* You may need to inspect the "Junk" or "Spam" folder of your mail application.
# Follow the instructions in that email.
#* Certain steps are needed only once because they pertain to your institution, notably filing a User Agreement.
#* Before work on a new proposal may begin, we may ask you to submit ''Activity Reports'' that are due for previous proposals upon their conclusion.
#* Fill in the ''Safety and Data form'', i.e., follow the ''User Work Submittal'' (UWS) link that is given in our notification email.
#:* Only the spokesperson may fill in the UWS form, or the PI may [mailto:cnm_useroffice@anl.gov?subject=Delegate%20UWS-PI%20submission designate a delegate]. Changes after the initial UWS submission by you (the PI or delegate) can only be made by your Scientific Contact at the CNM – see below.
#:* The UWS is '''due for each proposal''', as it collects certain metadata that we need, on a per-proposal-basis, for usage reporting to our funding agencies.
#:* Confirm or add users in section "Personnel Participating In User Proposal" of the UWS form. Enter the '''badge numbers''' for all participating users, as far as they are known at submission time.
# Direct all users who you expect to access ''Carbon'', possibly including yourself, to follow the instructions in section '''[[#Ongoing user access requirements]]'''.
# Be patient. '''UWS Processing typically takes at least one business day,''' as both our User Office staff and your Scientific Contact need to attend to your submission in person.
# The User Office will notify the spokesperson or delegate once work on a proposal may begin, issuing what we call the ''User Work Approval (UWA)''. It will be valid for one year for regular proposals, and 6 months for rapid-access proposals.

=== To get started as participating user ===
For new proposals or users new to CNM:
# Gently remind your spokesperson/PI to submit the UWS as shown above, or ask this person to nominate you via an email to the [mailto:cnm_useroffice@anl.gov?subject=Delegate%20UWS CNM User Office].
# Continue at section [[#Actions required by the User]].

For proposals already in progress, see the next section.

== Adding users to a proposal ==
Most users are nominated to work on a proposal when it is first submitted.
Additional users can be authorized to work under a proposal at any time after acceptance.
Actions are needed from both the user and the proposal spokesperson.

=== Actions required by the User ===
:1. [[#Ongoing user access requirements|Enter or review your registration]] as facility user and update as necessary.
:2. Follow up in a day or a week as specified at [[#Follow-up actions required by the User]].

=== Actions required by the PI or spokesperson ===
:3. Confirm that the user has entered or reviewed their registration, and determine their badge number(s).
:4. Add the user to the proposal. 
:* If the proposal has '''not yet started:''' See the section above [[#To get started as spokesperson or delegate]].
:* If the proposal has '''already started:''' Email the users' names '''and badge numbers''' to the proposal's ''Scientific Contact'' at the CNM and ask for the UWS to be augmented.


=== Actions required by the Scientific Contact ===
: 5. Open the UWS-SciCon form in the CNM Proposal Dashboard on the "Inside CNM" web page.
:* Add users under section "Participating Personnel".
:* Enter users' badge numbers which may have been left empty in earlier UWS submissions, as happens by necessity for very newly registered users.
:* Again, review that badge numbers are present and are correct for all relevant participants that are to work on ''Carbon'', or at least need to use {{sshgw}}.

=== Follow-up actions required by the User ===
: 6. Contact our [mailto:cnm_useroffice@anl.gov?subject=Process%20new%20user%20registration CNM User Office] and ask to review your registration and to initiate the next steps.
:* To work onsite at CNM or remotely on Carbon, you will need an ''Argonne computer account'', which is different from the account you used to enter your User Registration details – see section [[#Account Types]].
:* If you are new to Argonne, our User Office will be in contact with you to set up this account. Since the Argonne account is for more general use than your registration account, it will involve formal approvals, cyber security training, etc. and typically takes about ''one week'' to complete.
: 7. Complete or refresh training courses, as requested by the User Office.
:* Course requirements are triggered at various levels. There are courses for being a visitor at Argonne, a user at CNM, and for work performed under proposals, be it onsite or remotely.
: 8. Have your Argonne computer account finalized and ready.
:* '''New users:''' See section [[#Finalize your Argonne computer account]].
:* ''' Existing users:''' Check for and if necessary renew your expired password, [[#Verify or change your password|see below]].

=== Carbon account and access ===
When all prerequisites are met, your Argonne account will be enabled or (if so needed) re-enabled for use on Carbon.

: Enabling your account on Carbon ''does not happen immediately'' after a password reset or rename, or submitting proposal forms – it is a separate process.

: 9. '''Await confirmation''' from us that your account has been enabled ''on Carbon'' and been added to one or more proposals.
: 10. Review and follow the instructions on [[HPC/Network Access]], i.e., how to connect to the cluster.

If you do not receive an expected notification within two days, contact our [mailto:cnm_useroffice@anl.gov?subject=Process%20new%20user%20registration User Office] and describe your issue.

== Finalize your Argonne computer account ==
After your registration as a new Argonne Facility User has been processed, an [[#Account Types|Argonne "domain account"]] will be created for you.
Its user name will be in the form <code>b123456</code> and our User Office will have communicated to you an initial password.

To begin work on ''Carbon'' and possibly other CNM instruments, personalize both your ''password'' and your ''user name'', in this order.

; For Argonne personnel<nowiki>:</nowiki> User name vs. email address:
:* Your '''user name''' is the one by which you log in to Argonne-internal services, and takes one of three forms. Web login services typically indicate which user name form they expect..
:**<code>username</code> – This is the most typical form expected, used to log in to {{sshgw}}.
:** <code>username@ANL.GOV</code> – Your user name with a ''directory domain'' appended.
::* <code>ANL.GOV\username</code> – An alternate form of your user name and directory domain.
:* Your user name is '''not necessarily''' merely the name part of an Argonne '''email address''' that you will have if you work or study at Argonne.



=== Change your password ===
* Change your password as instructed when your account got created.
*: See also section [[#Verify or change your password]] below.
* If you chose to call the Argonne Help Desk at [tel:+1-630-252-9999 +1-630-252-9999], option 4, to make the password change:
*: While on the phone regarding your password, state that you are a ''CNM Carbon User'' and ask to also ''personalize your user name''.

=== Change your user name ===
# Request a user name change, either over the phone or by email. We will choose the user name for you.
#; To request by phone: Call the Argonne Help Desk at [tel:+1-630-252-9999 +1-630-252-9999], and select option 4.
#; To request by email: Follow this link: [mailto:help@anl.gov?subject=User%20name%20change%20request&body=I%20am%20a%20new%20user%20of%20CNM%27s%20Carbon%20HPC%20system.%20Please%20personalize%20the%20user%20name%20of%20my%20Argonne%20Domain%20computer%20account.%0D%0A%0D%0AMy%20full%20name%3A%09__%0D%0AArgonne%20Badge%3A%09__%0D%0A%0D%0AThank%20you.%0D%0A%0D%0A '''Open email template'''] – (Use a web browser that supports [https://en.wikipedia.org/wiki/Mailto "mailto:" links].)

## Respond from the ''same email address'' as you filled in at the CNM User Registration form.
##: In your email application, choose or switch to the ''mail account'' that handles the address you registered with at our User Registration. Messages pertaining to your Argonne activities that are sent from an unrelated email account may be ''disregarded'' (not taken up) by our help desk staff, for obvious reasons.
## Fill in your ''name'' and ''badge number'' in the email template that opens, then send the email.



# '''Await an emailed ticket number''' from Argonne's service desk, or '''follow the instructions of the person''' you called.
#* Ticket numbers are assigned upon manual review, typically within an hour or so during business hours.
#* If you '''did not get a ticket number''' after such reasonable time, re-send the request, making sure you do so ''from the email account you provided at User Registration''. 
# Check your email for follow-up questions and respond.
# Await the completion notice for your user name change.

=== Next steps ===
''Carbon'' access ''is not immediately enabled'' once your user name change request has been completed – it is a separate process.
# Expect an email that your ''Carbon'' access has been granted.

# Proceed under [[#Carbon account and access]].

== Proposal troubleshooting ==
'''Email the [mailto:cnm_useroffice@anl.gov?subject=Proposal%20Question CNM User Office]''' with any questions or concerns, such as:
* about proposals,
* about users, or
* you find that responding to any of your submissions or previous communications takes longer than a few business days.

'''Only our User Office staff''' is able to review all aspects of your proposal or your user access requirements, and determine any steps that have yet to be taken or need to be refreshed.

Include in your message:
* Proposal number(s),
* Name of the Principal Investigator (PI),
* Names and badge numbers of participants.

== Ongoing user access requirements ==
For you to log into Carbon and its SSH gateway, a number of criteria must be met, most of which are subject to '''expiration dates''' and '''require action from you for renewal'''.

You will ''not'' get notified (for various reasons) on some of these expirations.
Different duration terms may cause expirations to happen right in the middle of your proposal's lifecycle and can cause immediate inconvenience.
Depending on which criterion expired, one or both of the following will happen to your Argonne computer account:
* The account's permission to use {{sshgw}} will be revoked.
* The account ''as such'' will be disabled, equivalent to revoking all its permissions.

To recover, do one or more of the following:

=== User Registration ===
# Review and update your [https://beam.aps.anl.gov/pls/apsweb/ufr_main_pkg.usr_start_page '''User Registration.''']
#* For this system, log in with your ''badge number'' and the password associated with it, not your regular Argonne account – see section [[#Account Types]] above.
#* The registration itself requires renewal at least every 2 years, or earlier when any one of various prerequisite items expires.
#* For '''non-US citizens''' present in the US: Review and update registration items regarding your '''US visa''' or related work permit. A current US work or study status is required to access Argonne computers, the same as if you were to visit in person.
# Ask our [mailto:cnm_useroffice@anl.gov?subject=Reinstate%20user%20account CNM User Office] staff to review your registration updates and to recreate or re-instate your Argonne account, as the case might be.

=== Legal User Agreement ===
* If you '''changed your affiliation''' (the institution where you work or study), check to see if your new institution has a legal [https://aps.anl.gov/Users-Information/Legal-Financial/Argonne-User-Facility-Agreements '''User Facility Agreement'''] with Argonne in place and request one if not.
*: Be advised that the process may take several weeks to percolate through legal and adminstrative channels.

=== Training ===
*Review and renew your [http://beam.aps.anl.gov/pls/apsweb/rt0004.intro_process '''User Courses.''']
*: For remote users, the ESH223 course "Cybersecurity Annual Education and Awareness" is the one most likely to be in need of renewal.

=== Password ===
* Review and update the [[#Verify or change your password|'''password''']] to your Argonne computer account if you now have or previously had such an account.
*: If your account's password expired, we consider the account dormant and ''will not notify you about new proposals'' where you are listed as a participant.

=== Active proposal ===
You must be a participant in at least '''one active or recently expired [https://www.anl.gov/cnm/become-a-cnm-user User Proposal]'''.
You may run compute jobs under a given proposal while it is active, i.e., within the dates stated in the proposal's User Work Approval (UWA).
* To review the dates for your proposals, ask your PI to search their email archive for subjects ''"Work Approval Received"'' or ''"Proposal Expiration"''.

=== Data-only access for inactive proposals ===
After your last active proposal on Carbon has expired you may still access, for '''up to 30 days''', the SSH gateway and Carbon's login nodes.
Past that time window, your access will be revoked and '''your data may be deleted''', following [https://www.anl.gov/cnm/complete-your-work-at-cnm '''CNM's Data Retention Policy'''].

* While you still have access, offload from Carbon all your files and data that you may wish to keep.
*: The CNM cannot be expected or held responsible to store your data beyond your access window.

=== Practical hints ===
* Set yourself '''calendar entries''' about one year into the future to remind yourself to renew any of your user registration or training requirements.
* After you were added to a user proposal, '''wait at least an hour or more''' before trying to access our SSH gateway, preferably until the next morning.
*: Updates of your status need to be propagated through a handful of systems, each being done about hourly, so it may take several cycles for your status change to reach {{sshgw}}.

== Login issues ==
When you ask "I cannot log in" or "My password does not work", consider the following sections:

=== Review your access requirements ===
See section [[#Ongoing user access requirements]] above.

=== Verify or change your password ===
Do '''one of the following:'''
* Visit '''https://servicenow.anl.gov/pr''' 
** Complete the ''Password Enrollment'' process at this service on your first visit when you ''know'' your password. You will be (or have been) thus instructed when your account has been created.
** You can ''verify'' or ''change'' your existing known password.
** You can ''reset'' or ''unlock'' your password '''only''' after you enrolled.

** You cannot use this service if your account has been ''disabled'' as such, in which case revisit the section [[#Ongoing user access requirements]].
* You can ''change'' your ''known'' password also at our SSH gateway host <code>mega</code>.

** At least one of your CNM proposals must still be active.
** You cannot log in if your account is ''disabled'' as such.
: Error messages for login failures will not disclose a specific reason. You cannot tell if merely your password was wrong or your account is disabled.
* If you cannot use any of the above systems:
** request a [mailto:help@anl.gov?subject=Password%20reset password reset at the Argonne Service Desk], or
** request a [mailto:cnm_useroffice@anl.gov?subject=Requesting%20password%20reset password reset from the CNM User Office].


After a password update, you should be able to connect to {{sshgw}} normally.
If you find that you cannot log in, however, retry only '''after an hour or two''', as in some cases extra time is needed for passwords or other account changes to propagate.

=== Password lockouts ===
After trying incorrect passwords several times, your account may be '''temporarily locked out.''' To recover, try one of the following:
* Wait '''approximately 30 min.'''
* Unlock at '''https://servicenow.anl.gov/pr''' – available if you have previously enrolled there.

=== Review host names ===
Connect to the correct host names:
* When connecting from outside Argonne, at least ''two ssh sessions'' are required.
** {{sshgw|fqdn}} – an ssh "tunnel setup" connection.
** {{host|carbon}} – one or more "payload" connections for the ssh, scp, or sftp commands. Here, {{host|carbon}} is not a hostname but an entry in your ssh configuration that you must make (called a host alias or "profile", depending on the ssh application). The entry stands for a connection to localhost (your machine), at a port number forwarded by the preceding tunnel setup connection.
* When onsite (for any user) or using VPN (for Argonne staff):
** {{host|carbon.cnm.anl.gov}}. The previous name was {{host|clogin. …}}.
To learn more, read [[HPC/Network Access]].

=== Review network configuration ===
* Read again [[HPC/Network Access]], and follow the instructions for your platform.

=== Request support ===
* If the above sections did not help you to resolve your issue, [[HPC/Support|submit a support request]].

== Expected files or commands are unavailable ==
When you cannot create or edit files or directories under your home directory, or you find that commands you expect are not found, then:

# Inspect the beginning of your command line, called the ''prompt'':
#* If your prompt mentions "@mega" or "GATEWAY", e.g.:
#:: <code>''username'''''@mega5 CNM-GATEWAY''' $ </code>
#: then you are working '''not on Carbon just yet''', but rather on its SSH gateway. Visit [[HPC/Network_Access]] to learn how to connect properly.
#* If the prompt mentions "@login", e.g.:
#:: <code>''username'''''@login6''' … $</code>
#: then there may be a temporary file system issue which may resolve after a few minutes.
#: If such issue persists or recurs over several hours, however, read the message that will have appeared right after you logged in, above the first command prompt. To locate these messages, scroll up the content of your terminal application window, or log out and log back in to bring up these messages again.

=== My home directory is read-only ===
=== I cannot edit my .bashrc file ===
=== Expected commands unavailable ===
In all these situations, see [[#Expected files or commands are unavailable | the beginning of the current section]].

== Mailing lists ==
Announcements about Carbon are made on the [https://lists.anl.gov/mailman/listinfo/cnm-hpc-announce ''cnm-hpc-announce'' mailing list], hosted at Argonne.
The mailing list home page and its archive are, unfortunately, only accessible from onsite or (for authorized users) over VPN.

* To '''unsubscribe''' from the mailing list, do one of the following:
*# Open the "unsubscribe" link at the bottom of a recent message that you received from the mailing list.
*# Send a blank message to [mailto:cnm-hpc-announce-leave@lists.anl.gov cnm-hpc-announce'''-leave'''@lists.anl.gov] and follow up on the confirmation notice.
* To '''subscribe''' to the mailing list, send a blank message to [mailto:cnm-hpc-announce-join@lists.anl.gov cnm-hpc-announce'''-join'''@lists.anl.gov] and follow up on the confirmation notice.
* To '''change your email address''' on the list, simply do both of the above, in order.

; Hints:
* When sending commands by email, be sure to have the relevant email account selected in the "From" line of the compose window in your email application.
* Inspect the '''Junk''' mail folder of your email application if you do not receive a confirmation message for subscribing or unsubscribing within a minute or so.
* See the [http://www.gnu.org/software/mailman/mailman-member/node22.html GNU Mailman] documenation for background.

== Applications ==

=== I'd like to use application X ===
; Check if the application is already available on Carbon:
Either:
* Browse the [[HPC/Applications | Application Catalog]], or
* View the catalog on the Carbon command line:
<source lang="bash">
module avail
module -l avail 2>&1 | less
</source>
: The second form gives you browsable output.

; If you cannot find the application on Carbon:
* Submit a [[HPC/Support| support request]].
** Provide one or more ''URLs'' relevant to software you have in mind – be specific.
** Describe ''the problem'' you are trying to solve – it may well be that we can suggest an alternative solution.
** Give the extent of your planned use.
; If you see the application on Carbon but you cannot access it:
* Existing license agreements may cover only a subset of users (typically Argonne employees).
* If you feel you are eligible, submit a [[HPC/Support| support request]].
; If a version newer than the installed one on Carbon is available :
* Submit a [[HPC/Support| support request]].
* Include a URL to information about the new version.

===How do I run application X? ===
* [[HPC/Module Setup#Shell Customizations|Customize your shell environment]] to load the application module.
* Learn about [[HPC/Module Setup#Module Conventions on Carbon | module conventions on Carbon]].
* To determine the names of a package's executable scripts and binaries, load the application module (if you have not yet done so in your shell setup), then inspect the module's <code>$''NAME''_HOME/bin</code> directory. For instance, for the Quantum-ESPRESSO package:
<source lang="bash">
module load quantum_espresso
ls $QUANTUM_ESPRESSO_HOME/bin
</source>
* Learn how to [[HPC/Submitting and Managing Jobs | submit and manage jobs]].

=== How do I ''use'' application X? ===
Read the package's documentation, using one or more of the following:
* Inspect the package's <code>$''NAME''_HOME/share</code> or <code>$''NAME''_HOME/doc</code> directory on Carbon (see [[HPC/Module Setup#Module Conventions on Carbon | module conventions]]).
* Browse the package's web page, generally mentioned in the <code>module help</code> text or the [[HPC/Applications | application catalog]] entry.
* Consult a package's man pages. Few packages have them. Man page files are generally installed under <code>$''NAME''_HOME/man</code> or <code>$''NAME''_HOME/share/man</code> and if so, will be made available automatically to the <code>man</code> command.

== What's my account balance? ==
=== Simple answer: mybalance ===
To find out how many core-hours you have available, the simplest command to run is:
<source lang="bash">
mybalance -h
</source>
Project Machines Balance
-------- -------- ----------
user ANY 993.26
cnm34567 ANY 158760.93
cnm31234 ANY -148893.62
The table gives all the <code>Project</code>s you have access to (for use with the <code>qsub -A</code> argument), and their balance.
<code>Machine</code> lists all systems that can book jobs against your allocations. ''Carbon'' is currently the only machine that can do so.
<code>Balance</code> is your account balance, in ''core-hours'', as selected by the -h command option. This is the most useful and recommended unit.
Without -h, you get core-seconds, which are integers but rather more unwieldy numbers.
* The "user" project provides you with a small initial startup allocation of typically 1000 core-hours.
* When a Balance is reported as negative, that account typically has a CreditLimit assigned, which permits the balance to dip below zero. These details, however, are not shown by <code>mybalance</code>.

=== Complete answer: gbalance ===
To get allocation details for accounts that have CreditLimits, run the <code>gbalance</code> command. Pass on -u ''username'' or ''-p projectname'' to select your allocations:
gbalance -h -u $USER
: Type <code>$USER</code> ''as shown''. The command interpreter will fill in your actual username.
The ouput looks like:
Id Name Amount Reserved Balance CreditLimit '''Available'''
--- -------- ---------- -------- ---------- ----------- ---------
100 cnm31234 -148893.62 0.00 -148893.62 150000.00 '''1106.38'''
217 kpelzer 993.26 0.00 993.26 0.00 993.26
123 cnm34567 166440.93 7680.00 158760.93 0.00 158760.93
The most relevant column for you is '''Available'''. The units, given the -h option, are again core-hours.

The colums and their meanings are:
; Id: an internal number for the account.
; Name: The project name (for use with <code>qsub -A</code> or <code>#PBS -A</code>).
; Amount: Amount for transactions completely on the books for the project account; does not include running jobs or credits. Deposits are allocated by the User Office and implemented by the Carbon administrator.
; Reserved: Amounts held in reserve by all running jobs using this account. The reserve ensures that a job does not cause an overdraft when it finishes and when its actual use will be booked. The quantity is calculated by walltime * number of cores blocked. When a job terminates, the charge according to the actual time used will be ''subtracted'' from Amount, and the unused quantities will be ''re-added'' to Available.
; Balance: Available for new jobs; may go negative if CreditLimits are in place.
Balance = Amount - Reserved
; CreditLimit: Amount by which Balance may go negative; assigned by the Carbon administrator.
; Available: Relevant quantity for new jobs. Must be positive for a new job to start, and large enough to Reserve the entire job.
Available = Balance + CreditLimit

== Why did my account balance suddenly drop? ==
The compute time physically available by Carbon's processors is a "perishable" resource — CPU-hours pass by regardless of being used or not.
Therefore, your allocations are time-restricted in a "use-it-or-lose-it" manner.
This is done to encourage consistent use of the machine throughout allocation cycles.

Otherwise, resource contention becomes likely near the end of an allocation cycle, one year after each bunch of proposals has started, during which it would be increasingly impractical and eventually physically impossible to use up a large remaining allocation within a short time.
You would need to claim a large fraction of Carbon's nodes during a relatively short time window, which is unlikely to be possible because jobs from other users will be running as well.

=== Allocation expiration schedule ===
Unused allocations ''will expire'', following to schedule:
* Your allocation is provided in three equal-sized installments.
* All installments are active from the beginning.
* Installments ''expire'' in a staggered fashion, currently after 4, 8, and 12 months, respectively. A diagram might illuminate this:

Proposal Proposal
start expiration
|-------|-------|-------|------> Time
0 4 8 12 (months)
|
Installment
|
(1)|########.................... KEY
|
(2)|################............ . Installment is inactive
|
(3)|########################.... # Installment is active
|

* Your jobs will, sensibly, be booked against the installments that expire the earliest.

== My question is not answered here ==
See [[HPC/Support]].

HPC/FAQ

2025-09-19T17:14:25Z

Stern: /* Verify or change your password */

== Access and proposals ==
The HPC ''Carbon'' cluster and its associated software licenses may be accessed,
like any tool at the CNM, by one or both of the following routes:
* CNM Facility Users are eligible under [https://www.anl.gov/cnm/become-a-cnm-user '''CNM's User Access Program''']. Follow the link to learn how to initiate access under this route. Begin by registering as Facility User, then submit a proposal.
* Affiliates of [http://www.anl.gov/cnm Argonne's NST division] are eligible for discretionary access. Affiliate here refers to regular employees, Argonne/University joint appointees, visitor appointees, and student group members, being supervised by CNM staff.

The following is a summary of the main points as they pertain to ''Carbon''.

=== Legal User Agreement ===
The institution where you work or study must have a legal signed
[https://www.anl.gov/cnm/prepare-for-your-arrival '''User Facility Agreement''']
with Argonne in place before any work under an eventual proposal may be performed.

[https://aps.anl.gov/Users-Information/Legal-Financial/Argonne-User-Facility-Agreements '''Look up existing User Facility Agreements'''] and locate your institution if you have never worked with Argonne before, or you recently changed your affiliation, or are going to change it soon.

If your institution is not shown, ''get started'' with establishing a user agreement as soon as possible. Legal wheels turn slowly and your scientific work could be delayed.

=== User Registration ===
Under CNM's User Access Program, your starting point is to [https://www.anl.gov/cnm/how-to-register-to-become-a-cnm-user '''become a CNM user'''].

=== Proposal Lifecycle Overview ===

# To '''submit a proposal''', follow the instructions at the CNM's [https://www.anl.gov/cnm/cnm-user-information '''Call for Proposals'''].
# Upon successful review of a proposal and grant of an allocation, '''the contacting author''' will receive an email with subject '''User Proposal Status Notification'''.
# '''The contacting author must submit,''' for each proposal, a Safety&Data Form as instructed in the status email. This will confirm the names and badge numbers of participating users, as well as various characteristics of the proposal that we need for statistical reporting to our funding agencies.
# Participating '''users register and respond''' at various steps to have their Argonne user accounts created, activated or reactivated as needed, and have them authorized to access Carbon.
# Participating users '''perform the work proposed'''.
# [https://www.anl.gov/cnm/cnm-acknowledgment-statements-for-publications '''Credit CNM in publications'''] that result from your proposal work with us.
# After the proposal expires, our User Office will ask you to please submit:
#* An approximately 1-page '''Activity Report''' summarizing work performed and perhaps already published under the proposal. We use these reports to document the use of our facilities to funding agencies.
#* An end-of-proposal '''Satisfaction Survey''', to invite your feedback on our process.

== Account Types ==
You access our computer systems by two different user accounts:

=== User Registration Account ===
: – Looks like: <code>123456</code> – 
Your ''User Registration Account:''
* … is created when you register as CNM facility user, be it as participant or as Principal Investigator (PI), at the [https://beam.aps.anl.gov/pls/apsweb/ufr_main_pkg.usr_start_page '''User Registration'''] for the Advanced Photon Source (APS) and CNM.
* … has an all-numeric user name (no letters, no leading zeroes), which never changes – this is in fact your ''Argonne Badge Number'',
* … has its own password and [https://beam.aps.anl.gov/pls/apsweb/forgot_password.start_process '''password reset process at APS'''],

Use your User Registration Account to:
* … [https://beam.aps.anl.gov/pls/apsweb/ufr_main_pkg.usr_start_page '''update your user registration'''], such as when you change your affiliation, or to update visa information if Argonne needs them for you,
* … [https://pico.cnm.anl.gov/proposals/people/login '''submit a user proposal to CNM''']
* … [https://beam.aps.anl.gov/pls/apsweb/rt0004.intro_process '''take remote training'''] prior to beginning work, and prior to being assigned an Argonne domain account, which is described next.

=== Argonne Domain Account ===
: – Looks like: <code>'''b'''''123456''</code> or <code>''jsmith''</code> – 

Your ''Argonne Domain Account:''
* … is created '''when you first need it''', such as when you are a PI and have received the go-ahead on your first proposal with us,
* … has, at first, a user name beginning with <code>'''b'''</code>, followed by your badge number (without any leading zeroes),
* … will get, following your request, a personalized user name, such as <code>''jsmith''</code>
* … has its own password and uses its own [https://mypassword.anl.gov/ '''password reset and recovery''']. 

Use your Argonne Domain Account to:
* … submit the '''Safety&Data Form (UWS)''' of each proposal for which you are the PI or delegated spokesperson,
* … perform work – always remotely – at [[HPC|CNM's '''High Performance Computing system ''Carbon''''']], 
* … handle almost all other tasks at Argonne.

For assistance, please contact our [mailto:cnm_useroffice@anl.gov?subject=Account%20Question CNM User Office]. – Specify ''which task'' you have difficulties with.

== Getting started on a proposal ==

To start working on a proposal once it has been granted, '''action is required for each proposal''' by the proposal spokesperson (usually the Principal Investigator,PI), regardless of any previous proposals with us.
At this stage, we collect and update information about each proposal for reporting purposes to our funding agencies, and ensure that safety and training requirements are met by all participating users.

=== To get started as spokesperson or delegate ===
# Locate the email to you with subject ''User Proposal Status Notification'', sent by the CNM User Office.
#* If you are delayed in starting your proposal, you may need to expand your mailbox search time frame, to begin the search at possibly several months before present.
#* You may need to inspect the "Junk" or "Spam" folder of your mail application.
# Follow the instructions in that email.
#* Certain steps are needed only once because they pertain to your institution, notably filing a User Agreement.
#* Before work on a new proposal may begin, we may ask you to submit ''Activity Reports'' that are due for previous proposals upon their conclusion.
#* Fill in the ''Safety and Data form'', i.e., follow the ''User Work Submittal'' (UWS) link that is given in our notification email.
#:* Only the spokesperson may fill in the UWS form, or the PI may [mailto:cnm_useroffice@anl.gov?subject=Delegate%20UWS-PI%20submission designate a delegate]. Changes after the initial UWS submission by you (the PI or delegate) can only be made by your Scientific Contact at the CNM – see below.
#:* The UWS is '''due for each proposal''', as it collects certain metadata that we need, on a per-proposal-basis, for usage reporting to our funding agencies.
#:* Confirm or add users in section "Personnel Participating In User Proposal" of the UWS form. Enter the '''badge numbers''' for all participating users, as far as they are known at submission time.
# Direct all users who you expect to access ''Carbon'', possibly including yourself, to follow the instructions in section '''[[#Ongoing user access requirements]]'''.
# Be patient. '''UWS Processing typically takes at least one business day,''' as both our User Office staff and your Scientific Contact need to attend to your submission in person.
# The User Office will notify the spokesperson or delegate once work on a proposal may begin, issuing what we call the ''User Work Approval (UWA)''. It will be valid for one year for regular proposals, and 6 months for rapid-access proposals.

=== To get started as participating user ===
For new proposals or users new to CNM:
# Gently remind your spokesperson/PI to submit the UWS as shown above, or ask this person to nominate you via an email to the [mailto:cnm_useroffice@anl.gov?subject=Delegate%20UWS CNM User Office].
# Continue at section [[#Actions required by the User]].

For proposals already in progress, see the next section.

== Adding users to a proposal ==
Most users are nominated to work on a proposal when it is first submitted.
Additional users can be authorized to work under a proposal at any time after acceptance.
Actions are needed from both the user and the proposal spokesperson.

=== Actions required by the User ===
:1. [[#Ongoing user access requirements|Enter or review your registration]] as facility user and update as necessary.
:2. Follow up in a day or a week as specified at [[#Follow-up actions required by the User]].

=== Actions required by the PI or spokesperson ===
:3. Confirm that the user has entered or reviewed their registration, and determine their badge number(s).
:4. Add the user to the proposal. 
:* If the proposal has '''not yet started:''' See the section above [[#To get started as spokesperson or delegate]].
:* If the proposal has '''already started:''' Email the users' names '''and badge numbers''' to the proposal's ''Scientific Contact'' at the CNM and ask for the UWS to be augmented.


=== Actions required by the Scientific Contact ===
: 5. Open the UWS-SciCon form in the CNM Proposal Dashboard on the "Inside CNM" web page.
:* Add users under section "Participating Personnel".
:* Enter users' badge numbers which may have been left empty in earlier UWS submissions, as happens by necessity for very newly registered users.
:* Again, review that badge numbers are present and are correct for all relevant participants that are to work on ''Carbon'', or at least need to use {{sshgw}}.

=== Follow-up actions required by the User ===
: 6. Contact our [mailto:cnm_useroffice@anl.gov?subject=Process%20new%20user%20registration CNM User Office] and ask to review your registration and to initiate the next steps.
:* To work onsite at CNM or remotely on Carbon, you will need an ''Argonne computer account'', which is different from the account you used to enter your User Registration details – see section [[#Account Types]].
:* If you are new to Argonne, our User Office will be in contact with you to set up this account. Since the Argonne account is for more general use than your registration account, it will involve formal approvals, cyber security training, etc. and typically takes about ''one week'' to complete.
: 7. Complete or refresh training courses, as requested by the User Office.
:* Course requirements are triggered at various levels. There are courses for being a visitor at Argonne, a user at CNM, and for work performed under proposals, be it onsite or remotely.
: 8. Have your Argonne computer account finalized and ready.
:* '''New users:''' See section [[#Finalize your Argonne computer account]].
:* ''' Existing users:''' Check for and if necessary renew your expired password, [[#Verify or change your password|see below]].

=== Carbon account and access ===
When all prerequisites are met, your Argonne account will be enabled or (if so needed) re-enabled for use on Carbon.

: Enabling your account on Carbon ''does not happen immediately'' after a password reset or rename, or submitting proposal forms – it is a separate process.

: 9. '''Await confirmation''' from us that your account has been enabled ''on Carbon'' and been added to one or more proposals.
: 10. Review and follow the instructions on [[HPC/Network Access]], i.e., how to connect to the cluster.

If you do not receive an expected notification within two days, contact our [mailto:cnm_useroffice@anl.gov?subject=Process%20new%20user%20registration User Office] and describe your issue.

== Finalize your Argonne computer account ==
After your registration as a new Argonne Facility User has been processed, an [[#Account Types|Argonne "domain account"]] will be created for you.
Its user name will be in the form <code>b123456</code> and our User Office will have communicated to you an initial password.

To begin work on ''Carbon'' and possibly other CNM instruments, personalize both your ''password'' and your ''user name'', in this order.

; For Argonne personnel<nowiki>:</nowiki> User name vs. email address:
:* Your '''user name''' is the one by which you log in to Argonne-internal services, and takes one of three forms. Web login services typically indicate which user name form they expect..
:**<code>username</code> – This is the most typical form expected, used to log in to {{sshgw}}.
:** <code>username@ANL.GOV</code> – Your user name with a ''directory domain'' appended.
::* <code>ANL.GOV\username</code> – An alternate form of your user name and directory domain.
:* Your user name is '''not necessarily''' merely the name part of an Argonne '''email address''' that you will have if you work or study at Argonne.



=== Change your password ===
* Change your password as instructed when your account got created.
*: See also section [[#Verify or change your password]] below.
* If you chose to call the Argonne Help Desk at [tel:+1-630-252-9999 +1-630-252-9999], option 4, to make the password change:
*: While on the phone regarding your password, state that you are a ''CNM Carbon User'' and ask to also ''personalize your user name''.

=== Change your user name ===
# Request a user name change, either over the phone or by email. We will choose the user name for you.
#; To request by phone: Call the Argonne Help Desk at [tel:+1-630-252-9999 +1-630-252-9999], and select option 4.
#; To request by email: Follow this link: [mailto:help@anl.gov?subject=User%20name%20change%20request&body=I%20am%20a%20new%20user%20of%20CNM%27s%20Carbon%20HPC%20system.%20Please%20personalize%20the%20user%20name%20of%20my%20Argonne%20Domain%20computer%20account.%0D%0A%0D%0AMy%20full%20name%3A%09__%0D%0AArgonne%20Badge%3A%09__%0D%0A%0D%0AThank%20you.%0D%0A%0D%0A '''Open email template'''] – (Use a web browser that supports [https://en.wikipedia.org/wiki/Mailto "mailto:" links].)

## Respond from the ''same email address'' as you filled in at the CNM User Registration form.
##: In your email application, choose or switch to the ''mail account'' that handles the address you registered with at our User Registration. Messages pertaining to your Argonne activities that are sent from an unrelated email account may be ''disregarded'' (not taken up) by our help desk staff, for obvious reasons.
## Fill in your ''name'' and ''badge number'' in the email template that opens, then send the email.



# '''Await an emailed ticket number''' from Argonne's service desk, or '''follow the instructions of the person''' you called.
#* Ticket numbers are assigned upon manual review, typically within an hour or so during business hours.
#* If you '''did not get a ticket number''' after such reasonable time, re-send the request, making sure you do so ''from the email account you provided at User Registration''. 
# Check your email for follow-up questions and respond.
# Await the completion notice for your user name change.

=== Next steps ===
''Carbon'' access ''is not immediately enabled'' once your user name change request has been completed – it is a separate process.
# Expect an email that your ''Carbon'' access has been granted.

# Proceed under [[#Carbon account and access]].

== Proposal troubleshooting ==
'''Email the [mailto:cnm_useroffice@anl.gov?subject=Proposal%20Question CNM User Office]''' with any questions or concerns, such as:
* about proposals,
* about users, or
* you find that responding to any of your submissions or previous communications takes longer than a few business days.

'''Only our User Office staff''' is able to review all aspects of your proposal or your user access requirements, and determine any steps that have yet to be taken or need to be refreshed.

Include in your message:
* Proposal number(s),
* Name of the Principal Investigator (PI),
* Names and badge numbers of participants.

== Ongoing user access requirements ==
For you to log into Carbon and its SSH gateway, a number of criteria must be met, most of which are subject to '''expiration dates''' and '''require action from you for renewal'''.

You will ''not'' get notified (for various reasons) on some of these expirations.
Different duration terms may cause expirations to happen right in the middle of your proposal's lifecycle and can cause immediate inconvenience.
Depending on which criterion expired, one or both of the following will happen to your Argonne computer account:
* The account's permission to use {{sshgw}} will be revoked.
* The account ''as such'' will be disabled, equivalent to revoking all its permissions.

To recover, do one or more of the following:

=== User Registration ===
# Review and update your [https://beam.aps.anl.gov/pls/apsweb/ufr_main_pkg.usr_start_page '''User Registration.''']
#* For this system, log in with your ''badge number'' and the password associated with it, not your regular Argonne account – see section [[#Account Types]] above.
#* The registration itself requires renewal at least every 2 years, or earlier when any one of various prerequisite items expires.
#* For '''non-US citizens''' present in the US: Review and update registration items regarding your '''US visa''' or related work permit. A current US work or study status is required to access Argonne computers, the same as if you were to visit in person.
# Ask our [mailto:cnm_useroffice@anl.gov?subject=Reinstate%20user%20account CNM User Office] staff to review your registration updates and to recreate or re-instate your Argonne account, as the case might be.

=== Legal User Agreement ===
* If you '''changed your affiliation''' (the institution where you work or study), check to see if your new institution has a legal [https://aps.anl.gov/Users-Information/Legal-Financial/Argonne-User-Facility-Agreements '''User Facility Agreement'''] with Argonne in place and request one if not.
*: Be advised that the process may take several weeks to percolate through legal and adminstrative channels.

=== Training ===
*Review and renew your [http://beam.aps.anl.gov/pls/apsweb/rt0004.intro_process '''User Courses.''']
*: For remote users, the ESH223 course "Cybersecurity Annual Education and Awareness" is the one most likely to be in need of renewal.

=== Password ===
* Review and update the [[#Verify or change your password|'''password''']] to your Argonne computer account if you now have or previously had such an account.
*: If your account's password expired, we consider the account dormant and ''will not notify you about new proposals'' where you are listed as a participant.

=== Active proposal ===
You must be a participant in at least '''one active or recently expired [https://www.anl.gov/cnm/become-a-cnm-user User Proposal]'''.
You may run compute jobs under a given proposal while it is active, i.e., within the dates stated in the proposal's User Work Approval (UWA).
* To review the dates for your proposals, ask your PI to search their email archive for subjects ''"Work Approval Received"'' or ''"Proposal Expiration"''.

=== Data-only access for inactive proposals ===
After your last active proposal on Carbon has expired you may still access, for '''up to 30 days''', the SSH gateway and Carbon's login nodes.
Past that time window, your access will be revoked and '''your data may be deleted''', following [https://www.anl.gov/cnm/complete-your-work-at-cnm '''CNM's Data Retention Policy'''].

* While you still have access, offload from Carbon all your files and data that you may wish to keep.
*: The CNM cannot be expected or held responsible to store your data beyond your access window.

=== Practical hints ===
* Set yourself '''calendar entries''' about one year into the future to remind yourself to renew any of your user registration or training requirements.
* After you were added to a user proposal, '''wait at least an hour or more''' before trying to access our SSH gateway, preferably until the next morning.
*: Updates of your status need to be propagated through a handful of systems, each being done about hourly, so it may take several cycles for your status change to reach {{sshgw}}.

== Login issues ==
When you ask "I cannot log in" or "My password does not work", consider the following sections:

=== Review your access requirements ===
See section [[#Ongoing user access requirements]] above.

=== Verify or change your password ===
Do '''one of the following:'''
* Visit '''https://servicenow.anl.gov/pr''' 
** Complete the ''Password Enrollment'' process at this service on your first visit when you ''know'' your password. You will be (or have been) thus instructed when your account has been created.
** You can ''verify'' or ''change'' your existing known password.
** You can ''reset'' or ''unlock'' your password '''only''' after you enrolled.

** You cannot use this service if your account has been ''disabled'' as such, in which case revisit the section [[#Ongoing user access requirements]].
* You can ''change'' your ''known'' password also at our SSH gateway host <code>mega</code>.

** At least one of your CNM proposals must still be active.
** You cannot log in if your account is ''disabled'' as such.
: Error messages for login failures will not disclose a specific reason. You cannot tell if merely your password was wrong or your account is disabled.
* If you cannot use any of the above systems:
** request a [mailto:help@anl.gov?subject=Password%20reset password reset at the Argonne Service Desk], or
** request a [mailto:cnm_useroffice@anl.gov?subject=Requesting%20password%20reset password reset from the CNM User Office].


After a password update, you should be able to connect to {{sshgw}} normally.
If you find that you cannot log in, however, retry only '''after an hour or two''', as in some cases extra time is needed for passwords or other account changes to propagate.

=== Password lockouts ===
After trying incorrect passwords several times, your account may be '''temporarily locked out.''' To recover, try one of the following:
* Wait '''approximately 30 min.'''
* Unlock at '''https://mypassword.anl.gov/''' – available if you have previously enrolled there.

=== Review host names ===
Connect to the correct host names:
* When connecting from outside Argonne, at least ''two ssh sessions'' are required.
** {{sshgw|fqdn}} – an ssh "tunnel setup" connection.
** {{host|carbon}} – one or more "payload" connections for the ssh, scp, or sftp commands. Here, {{host|carbon}} is not a hostname but an entry in your ssh configuration that you must make (called a host alias or "profile", depending on the ssh application). The entry stands for a connection to localhost (your machine), at a port number forwarded by the preceding tunnel setup connection.
* When onsite (for any user) or using VPN (for Argonne staff):
** {{host|carbon.cnm.anl.gov}}. The previous name was {{host|clogin. …}}.
To learn more, read [[HPC/Network Access]].

=== Review network configuration ===
* Read again [[HPC/Network Access]], and follow the instructions for your platform.

=== Request support ===
* If the above sections did not help you to resolve your issue, [[HPC/Support|submit a support request]].

== Expected files or commands are unavailable ==
When you cannot create or edit files or directories under your home directory, or you find that commands you expect are not found, then:

# Inspect the beginning of your command line, called the ''prompt'':
#* If your prompt mentions "@mega" or "GATEWAY", e.g.:
#:: <code>''username'''''@mega5 CNM-GATEWAY''' $ </code>
#: then you are working '''not on Carbon just yet''', but rather on its SSH gateway. Visit [[HPC/Network_Access]] to learn how to connect properly.
#* If the prompt mentions "@login", e.g.:
#:: <code>''username'''''@login6''' … $</code>
#: then there may be a temporary file system issue which may resolve after a few minutes.
#: If such issue persists or recurs over several hours, however, read the message that will have appeared right after you logged in, above the first command prompt. To locate these messages, scroll up the content of your terminal application window, or log out and log back in to bring up these messages again.

=== My home directory is read-only ===
=== I cannot edit my .bashrc file ===
=== Expected commands unavailable ===
In all these situations, see [[#Expected files or commands are unavailable | the beginning of the current section]].

== Mailing lists ==
Announcements about Carbon are made on the [https://lists.anl.gov/mailman/listinfo/cnm-hpc-announce ''cnm-hpc-announce'' mailing list], hosted at Argonne.
The mailing list home page and its archive are, unfortunately, only accessible from onsite or (for authorized users) over VPN.

* To '''unsubscribe''' from the mailing list, do one of the following:
*# Open the "unsubscribe" link at the bottom of a recent message that you received from the mailing list.
*# Send a blank message to [mailto:cnm-hpc-announce-leave@lists.anl.gov cnm-hpc-announce'''-leave'''@lists.anl.gov] and follow up on the confirmation notice.
* To '''subscribe''' to the mailing list, send a blank message to [mailto:cnm-hpc-announce-join@lists.anl.gov cnm-hpc-announce'''-join'''@lists.anl.gov] and follow up on the confirmation notice.
* To '''change your email address''' on the list, simply do both of the above, in order.

; Hints:
* When sending commands by email, be sure to have the relevant email account selected in the "From" line of the compose window in your email application.
* Inspect the '''Junk''' mail folder of your email application if you do not receive a confirmation message for subscribing or unsubscribing within a minute or so.
* See the [http://www.gnu.org/software/mailman/mailman-member/node22.html GNU Mailman] documenation for background.

== Applications ==

=== I'd like to use application X ===
; Check if the application is already available on Carbon:
Either:
* Browse the [[HPC/Applications | Application Catalog]], or
* View the catalog on the Carbon command line:
<source lang="bash">
module avail
module -l avail 2>&1 | less
</source>
: The second form gives you browsable output.

; If you cannot find the application on Carbon:
* Submit a [[HPC/Support| support request]].
** Provide one or more ''URLs'' relevant to software you have in mind – be specific.
** Describe ''the problem'' you are trying to solve – it may well be that we can suggest an alternative solution.
** Give the extent of your planned use.
; If you see the application on Carbon but you cannot access it:
* Existing license agreements may cover only a subset of users (typically Argonne employees).
* If you feel you are eligible, submit a [[HPC/Support| support request]].
; If a version newer than the installed one on Carbon is available :
* Submit a [[HPC/Support| support request]].
* Include a URL to information about the new version.

===How do I run application X? ===
* [[HPC/Module Setup#Shell Customizations|Customize your shell environment]] to load the application module.
* Learn about [[HPC/Module Setup#Module Conventions on Carbon | module conventions on Carbon]].
* To determine the names of a package's executable scripts and binaries, load the application module (if you have not yet done so in your shell setup), then inspect the module's <code>$''NAME''_HOME/bin</code> directory. For instance, for the Quantum-ESPRESSO package:
<source lang="bash">
module load quantum_espresso
ls $QUANTUM_ESPRESSO_HOME/bin
</source>
* Learn how to [[HPC/Submitting and Managing Jobs | submit and manage jobs]].

=== How do I ''use'' application X? ===
Read the package's documentation, using one or more of the following:
* Inspect the package's <code>$''NAME''_HOME/share</code> or <code>$''NAME''_HOME/doc</code> directory on Carbon (see [[HPC/Module Setup#Module Conventions on Carbon | module conventions]]).
* Browse the package's web page, generally mentioned in the <code>module help</code> text or the [[HPC/Applications | application catalog]] entry.
* Consult a package's man pages. Few packages have them. Man page files are generally installed under <code>$''NAME''_HOME/man</code> or <code>$''NAME''_HOME/share/man</code> and if so, will be made available automatically to the <code>man</code> command.

== What's my account balance? ==
=== Simple answer: mybalance ===
To find out how many core-hours you have available, the simplest command to run is:
<source lang="bash">
mybalance -h
</source>
Project Machines Balance
-------- -------- ----------
user ANY 993.26
cnm34567 ANY 158760.93
cnm31234 ANY -148893.62
The table gives all the <code>Project</code>s you have access to (for use with the <code>qsub -A</code> argument), and their balance.
<code>Machine</code> lists all systems that can book jobs against your allocations. ''Carbon'' is currently the only machine that can do so.
<code>Balance</code> is your account balance, in ''core-hours'', as selected by the -h command option. This is the most useful and recommended unit.
Without -h, you get core-seconds, which are integers but rather more unwieldy numbers.
* The "user" project provides you with a small initial startup allocation of typically 1000 core-hours.
* When a Balance is reported as negative, that account typically has a CreditLimit assigned, which permits the balance to dip below zero. These details, however, are not shown by <code>mybalance</code>.

=== Complete answer: gbalance ===
To get allocation details for accounts that have CreditLimits, run the <code>gbalance</code> command. Pass on -u ''username'' or ''-p projectname'' to select your allocations:
gbalance -h -u $USER
: Type <code>$USER</code> ''as shown''. The command interpreter will fill in your actual username.
The ouput looks like:
Id Name Amount Reserved Balance CreditLimit '''Available'''
--- -------- ---------- -------- ---------- ----------- ---------
100 cnm31234 -148893.62 0.00 -148893.62 150000.00 '''1106.38'''
217 kpelzer 993.26 0.00 993.26 0.00 993.26
123 cnm34567 166440.93 7680.00 158760.93 0.00 158760.93
The most relevant column for you is '''Available'''. The units, given the -h option, are again core-hours.

The colums and their meanings are:
; Id: an internal number for the account.
; Name: The project name (for use with <code>qsub -A</code> or <code>#PBS -A</code>).
; Amount: Amount for transactions completely on the books for the project account; does not include running jobs or credits. Deposits are allocated by the User Office and implemented by the Carbon administrator.
; Reserved: Amounts held in reserve by all running jobs using this account. The reserve ensures that a job does not cause an overdraft when it finishes and when its actual use will be booked. The quantity is calculated by walltime * number of cores blocked. When a job terminates, the charge according to the actual time used will be ''subtracted'' from Amount, and the unused quantities will be ''re-added'' to Available.
; Balance: Available for new jobs; may go negative if CreditLimits are in place.
Balance = Amount - Reserved
; CreditLimit: Amount by which Balance may go negative; assigned by the Carbon administrator.
; Available: Relevant quantity for new jobs. Must be positive for a new job to start, and large enough to Reserve the entire job.
Available = Balance + CreditLimit

== Why did my account balance suddenly drop? ==
The compute time physically available by Carbon's processors is a "perishable" resource — CPU-hours pass by regardless of being used or not.
Therefore, your allocations are time-restricted in a "use-it-or-lose-it" manner.
This is done to encourage consistent use of the machine throughout allocation cycles.

Otherwise, resource contention becomes likely near the end of an allocation cycle, one year after each bunch of proposals has started, during which it would be increasingly impractical and eventually physically impossible to use up a large remaining allocation within a short time.
You would need to claim a large fraction of Carbon's nodes during a relatively short time window, which is unlikely to be possible because jobs from other users will be running as well.

=== Allocation expiration schedule ===
Unused allocations ''will expire'', following to schedule:
* Your allocation is provided in three equal-sized installments.
* All installments are active from the beginning.
* Installments ''expire'' in a staggered fashion, currently after 4, 8, and 12 months, respectively. A diagram might illuminate this:

Proposal Proposal
start expiration
|-------|-------|-------|------> Time
0 4 8 12 (months)
|
Installment
|
(1)|########.................... KEY
|
(2)|################............ . Installment is inactive
|
(3)|########################.... # Installment is active
|

* Your jobs will, sensibly, be booked against the installments that expire the earliest.

== My question is not answered here ==
See [[HPC/Support]].

HPC/Applications/comsol

2025-08-11T21:18:49Z

Stern: /* License configuration when outside CNM and using SSH Tunneling */

== Introduction ==
The [http://www.comsol.com/products/multiphysics/ '''COMSOL Multiphysics'''] application suite
is available for Mac, Windows, and Linux.

== Access ==
[[Image:HPC Comsol licensed modules.png|right|450px|thumb|COMSOL features licensed at CNM as of 2024-10.]]
=== COMSOL Multiphysics ===
Access to '''COMSOL Multiphysics''', the main application with both a graphical user interface and batch computing capabilities, will be granted '''''upon request''''' .

You must be:
* An '''''Argonne employee'''''  in one of the following roles:
** '''''NST staff member'''''  (including PostDocs, joint appointments, and students at NST).
** '''''CNM Facility User'''''  (including ''non-NST Argonne employees''), with an active proposal that has COMSOL in scope. Please see [[HPC/FAQ#Access_and_proposals]].

The set of available COMSOL modules that we have licensed is shown at right.

* The native desktop version can be installed only on ''Argonne-owned'' computers (laptop or desktop). Please contact [mailto:help@anl.gov?Subject=COMSOL%20license%20access%20request CNM support]. Make sure to review [http://www.comsol.com/support/updates/ Product Updates].
* To run COMSOL on our HPC system ''Carbon'', as GUI or as batch computing jobs, your proposal must have ''Carbon'' access in scope.

=== COMSOL Server ===
''CNM Facility Users'' who are not Argonne employees are not eligible to access
the full '''COMSOL Multiphysics''' product, in compliance with its licensing terms and conditions.
However, the [https://www.comsol.com/comsol-server '''COMSOL Server'''] product offers an alternative
albeit more narrow means of access.

All users, including '''''non-Argonne users''''' may be able to run, over a web interface in their browser,
pre-parameterized "fixed-model" simulations that were made with the '''COMSOL Application Builder''' (requires Windows).

To inquire about using COMSOL server applications,
please contact us.

We will review possible purchase paths for COMSOL Server, and the ''considerable effort'' likely required by users and the CNM to develop and deploy a COMSOL application.

== Support ==
* For questions on how to run or start COMSOL:
*: Submit an [[HPC/Support | '''IT support request''']]
* For questions related to most COMSOL data or scripts, contact COMSOL support at:
*: [mailto:support@comsol.com '''support'''@comsol.com]
* '''Exception:''' If your support question might contain data that is subject to '''US Export Control''' regulations, contact instead:
*: [mailto:us-export-control@us.comsol.com '''us-export-control'''@'''us'''.comsol.com]

== The modes to run COMSOL ==
There are several ways of running COMSOL.

=== Running the desktop application ===
To edit models and analyze results, use the COMSOL graphical user interface (GUI). You can run it in several ways, each explained in more detail in the following sections.

# Run on Carbon, display on your desktop using the [http://en.wikipedia.org/wiki/X_Window_System X Window system (X11)].
#* Requires an ''X11 Server'' application running on your client machine, and SSH X11 forwarding.
#* The X11 protocol can be slow over LAN.
# Run on Carbon within a [[HPC/Network Access/Virtual Desktop | virtual desktop environment (VNC)]].
#* Requires a VNC client on your machine.
#* It can be confusing where VNC and the app runs.
#* The virtual desktop presents a (deliberatly) sparse windowing environment – Copy&Paste can be difficult to use.
# Run on your own computer, remotely checking out Carbon's license.
#* Requires a full installation of COMSOL on your desktoo or laptop computer.
#* You will require access to the COMSOL license server running on Carbon, either through VPN (fairly easy) or SSH tunneling (can be intimidating for first-time users). See below for details.

=== Running a batch calculation ===
When the computational stage of models takes longer than a few minutes to complete, it will be beneficial to submit save the model file and run it in parallel on Carbon.
See the [[#Batch jobs|section below]].

== Use with X11 ==
For casual use, and with [[HPC/Network Access/SSH Tunnel Setup on Linux and MacOS | X11 tunneling to Carbon]] set up, simply add to your <code>~/.bashrc</code> file on Carbon:
module load comsol

Then, from the command line on a Carbon login node, start the COMSOL GUI:
comsol &

License configuration is automatic on Carbon.

Comsol will use a number of cores to run calculations multithreaded by default. (On Carbon, COMSOL_NUM_THREADS is set.)

'''Quit the application''' as soon as you are done. Only one user can use Comsol at a time.

== Use with VNC ==
See general [[HPC/Network Access/Virtual Desktop | instructions on using VNC]], and the COMSOL example therein.

== Native desktop version ==

To get the fastest response from the COMSOL GUI, run a native COMSOL version on your desktop machine or laptop, called ''client machine'' below.

Enter license details either in the COMSOL Setup configuration dialog.

== Native desktop version — License configuration ==
A client's COMSOL license needs to be configured depending on how the client reaches CNM network resources. Follow the appropriate section:
* [[#License configuration when inside CNM or using VPN]]
* [[#License configuration when outside CNM and using SSH Tunneling]]

=== License configuration when inside CNM or using VPN ===
==== Ensure correct DNS search domains ====

* Inspect the client machine's network configuration.
* Ensure that '''two''' entries are listed as DNS Search Domains for the current network profile:
nst.anl.gov
cnm.anl.gov
: (The order of entries does not matter for the purposes considered here.)
* If the client machine is ''wired to CNM networks'', the domains are normally provided correctly by DHCP.
* If the client machine is ''connected over VPN'', you may need edit the respective network profile.
:* On a Mac, find the VPN configuration details under System Preferences, Network, Advanced… . You may need to unlock the Network panel first before you can make changes under "Advanced…". It appears that those changes are being saved and will be activated automatically the next time that this VPN session gets started.
: [[Image:VNC_add_cnm_to_DNS_1.png|400px]] [[Image:VNC_add_cnm_to_DNS_2.png|400px]]
:* On Windows or Linux machines, take ''analogous'' steps, using the platform's own configuration tools.

* Exclude license manager components from client desktop installation. (Those are all handled server-side.)

==== License format ====
* Floating Network License (FNL), "Three-server redundancy"

==== License servers ====
* On client machines, while running the COMSOL installer or license configuration, specify license servers and ports as follows:
clicense1 1718
clicense2 1718
clicense3 1718
: '''Note:''' Use ''short host name'' form exactly as given here. (Technical Reason: The license must be usable both on Carbon's internal network as well as on divisional networks that are outside of Carbon).

'''Advanced:''' A client machine's COMSOL license file <code><COMSOLDIR>/Multiphysics/license/license.dat</code> should look as follows:
<source lang="make">
SERVER clicense1 ANY 1718
SERVER clicense2 ANY 1718
SERVER clicense3 ANY 1718
USE_SERVER
</source>

<code><COMSOLDIR></code> is located by default at:
* '''Windows''': C:\Program files\COMSOL\COMSOL''nn''
* '''Linux''': /usr/local/comsol''nn''
* '''Mac''': /Applications/COMSOL''nn''
("''nn''" is the application's abbreviated version number.)

==== Network Troubleshooting ====
When using the license access model descibed here (as opposed to tunneled over SSH), ensure the following:
* Verify that short host names can be resolved to IP addresses. Either one of the following commands should produce sensible output:
host clicense1
: or:
ping -c 3 clicense1
* Verify that ''both license ports on the same server'' can be reached from your client machine:
telnet '''clicense1''' 1718
telnet '''clicense1''' 1719
: or:
telnet '''clicense2''' 1718
telnet '''clicense2''' 1719
: A ''successful'' connection looks like:
Trying 146.139.x.y...
Connected to clicense1.cnm.anl.gov.
Escape character is '^]'.
:: Do not enter any data. Instead, close each connection by hitting <code>Ctrl</code><code>]</code> (closing square bracket), followed by <code>Ctrl</code><code>D</code>.
: A ''failed'' connection gives one of these errors:
telnet: connect to address 146.139.x.y: Operation timed out
: or:
telnet: connect to address 146.139.x.y: Connection refused
:: In these cases, ask that your IP firewall configuration be reviewed and changed.

=== License access when outside CNM ===
If you're comfortable with ssh and tunneling, set up client licensing as follows:

==== Configure COMSOL ====
The following needs to be done just once, typically during COMSOL installation:

In the COMSOL Installer, or in COMSOL Preferences, enter:
* '''License format:''' Floating Network License (FNL), "Port number and Hostname"
:: "Three-server redundancy" cannot be used because is cannot (easily) be accessed over SSH. Configure with just one server, and in rare tweak SSH to tunnel to the other server.
* '''Server:''' <code>127.0.0.1</code>
:: Do not use "localhost", as COMSOL interprets this incorrectly to mean that your machine ''is'' a license server.

* '''Port:''' <code>1718</code>


==== Provide server connection ====
This step needs to be done prior to running COMSOL in your desktop session:
* Connect to <code>mega</code> with the following ssh command:
ssh -L 1718:'''clicense1''':1718 -L 1719:'''clicense1''':1719 mega.cnm.anl.gov
* Leave the command running between invocations of COMSOL.

==== Run COMSOL ====
* Ensure your connection to "mega" is still running. If not, restart it.
* Start the COMSOL application.

** If you get an error that the number of licenses is "exhausted", that is confirmation that you correctly reached the license server, though other users are using licenses.
** If the license access attempt takes a long time or hangs, the service might temporarily run on a secondary server. In this case, stop the mega session and restart it, connecting to the alternate license server:
ssh -L 1718:clicense'''2''':1718 -L 1719:clicense'''2''':1719 mega.cnm.anl.gov

== License Troubleshooting ==
The following are typical error messages.
=== Could not obtain license for COMSOL Multiphysics GUI ===
* Error details:
License error: -4.
'''Licensed number of users already reached.'''
Feature: COMSOLGUI
License path: /Users/stern/Applications/COMSOL52a/license/license.dat:
FlexNet Licensing error:-4,132
* Reason: The connection to the license server succeeded, but the license token(s) you requested are in use.
* Solution: Wait a bit, or [http://carbonmgmt/ganglia/lic.php identify who's using COMSOL] and politely ask the user when they might be finished with their work.

=== License server machine is down or not responding ===
* Error details:
License error: -96.
See the system administrator about starting the license server system, or
make sure you're referring to the right host (see LM_LICENSE_FILE).
Feature: SERIAL
Hostname: clicense3
License path: /Users/stern/Applications/COMSOL52a/license/license.dat:
FlexNet Licensing error:-96,7
* Reason: The license servers cannot be reached at all. The <code>ping</code> or <code>host</code> tests mentioned above fail.
* Solution: Review the client license file and DNS lookup domains.

=== License file does not support this version ===
* Error details:
Feature: SERIAL
'''Application version > License version: 5.21 > 3.5'''
License path: /Users/stern/Applications/COMSOL52a/license/license.dat:
FlexNet Licensing error:-21,126
* Reason: Only the first of two steps for connecting to the license servers succeeds, which may happen when the redundant license service runs on the alternate server instead of the one you're connected to.
* Solution: When using SSH, try <code>clicense2</code> instead of <code>clicense1</code>, or vice versa. Revisit your <code>~/.ssh/config</code> file or use the alternate command line as shown in the SSH section above.

== Multiple CPU cores ==
Ways to start multi-core (from http://www.comsol.com/support/knowledgebase/1096/)
comsol -np 8

export COMSOL_NUM_THREADS=8 # automatically provided on ''Carbon''
comsol
* COMSOL does not benefit from hyperthreading (oversubscribing the physical core count).

== Batch jobs ==

* http://www.comsol.com/support/knowledgebase/1001/ .
* [[Media:HPC 2012-05-16 Comsol parallel usage.pdf]] (Modified from http://www.comsol.com/support/knowledgebase/1001/files/4027/cluster_install_linux_42a.pptx – Appendix slide, PBS)
* '''Note:''' MPD is not supported on Carbon (I patched the main comsol script), and no longer necessary at all in Comsol-4.3.

=== Sample Job file ===
See:
$COMSOL_HOME/sample.job

== Important COMSOL Support Articles ==
* Toplevel suport page: https://www.comsol.com/support

=== KB Articles ===
* [https://www.comsol.com/support/knowledgebase/1001 Running COMSOL® in parallel on clusters]
* [https://www.comsol.com/support/knowledgebase/1250 Running parametric sweeps, batch sweeps, and cluster sweeps from the command line]
* [https://www.comsol.com/support/knowledgebase/1255 Reducing the amount of solution data stored in a model]
* [https://www.comsol.com/support/knowledgebase/830 Handling large models]

=== Blog-style articles ===
* [https://www.comsol.com/blogs/the-power-of-the-batch-sweep/ The Power of the Batch Sweep]
* [https://www.comsol.com/blogs/how-to-run-simulations-in-batch-mode-from-the-command-line/ How to Run Simulations in Batch Mode from the Command Line]

HPC/Applications/comsol

2025-08-11T20:59:12Z

Stern: /* License server */

== Introduction ==
The [http://www.comsol.com/products/multiphysics/ '''COMSOL Multiphysics'''] application suite
is available for Mac, Windows, and Linux.

== Access ==
[[Image:HPC Comsol licensed modules.png|right|450px|thumb|COMSOL features licensed at CNM as of 2024-10.]]
=== COMSOL Multiphysics ===
Access to '''COMSOL Multiphysics''', the main application with both a graphical user interface and batch computing capabilities, will be granted '''''upon request''''' .

You must be:
* An '''''Argonne employee'''''  in one of the following roles:
** '''''NST staff member'''''  (including PostDocs, joint appointments, and students at NST).
** '''''CNM Facility User'''''  (including ''non-NST Argonne employees''), with an active proposal that has COMSOL in scope. Please see [[HPC/FAQ#Access_and_proposals]].

The set of available COMSOL modules that we have licensed is shown at right.

* The native desktop version can be installed only on ''Argonne-owned'' computers (laptop or desktop). Please contact [mailto:help@anl.gov?Subject=COMSOL%20license%20access%20request CNM support]. Make sure to review [http://www.comsol.com/support/updates/ Product Updates].
* To run COMSOL on our HPC system ''Carbon'', as GUI or as batch computing jobs, your proposal must have ''Carbon'' access in scope.

=== COMSOL Server ===
''CNM Facility Users'' who are not Argonne employees are not eligible to access
the full '''COMSOL Multiphysics''' product, in compliance with its licensing terms and conditions.
However, the [https://www.comsol.com/comsol-server '''COMSOL Server'''] product offers an alternative
albeit more narrow means of access.

All users, including '''''non-Argonne users''''' may be able to run, over a web interface in their browser,
pre-parameterized "fixed-model" simulations that were made with the '''COMSOL Application Builder''' (requires Windows).

To inquire about using COMSOL server applications,
please contact us.

We will review possible purchase paths for COMSOL Server, and the ''considerable effort'' likely required by users and the CNM to develop and deploy a COMSOL application.

== Support ==
* For questions on how to run or start COMSOL:
*: Submit an [[HPC/Support | '''IT support request''']]
* For questions related to most COMSOL data or scripts, contact COMSOL support at:
*: [mailto:support@comsol.com '''support'''@comsol.com]
* '''Exception:''' If your support question might contain data that is subject to '''US Export Control''' regulations, contact instead:
*: [mailto:us-export-control@us.comsol.com '''us-export-control'''@'''us'''.comsol.com]

== The modes to run COMSOL ==
There are several ways of running COMSOL.

=== Running the desktop application ===
To edit models and analyze results, use the COMSOL graphical user interface (GUI). You can run it in several ways, each explained in more detail in the following sections.

# Run on Carbon, display on your desktop using the [http://en.wikipedia.org/wiki/X_Window_System X Window system (X11)].
#* Requires an ''X11 Server'' application running on your client machine, and SSH X11 forwarding.
#* The X11 protocol can be slow over LAN.
# Run on Carbon within a [[HPC/Network Access/Virtual Desktop | virtual desktop environment (VNC)]].
#* Requires a VNC client on your machine.
#* It can be confusing where VNC and the app runs.
#* The virtual desktop presents a (deliberatly) sparse windowing environment – Copy&Paste can be difficult to use.
# Run on your own computer, remotely checking out Carbon's license.
#* Requires a full installation of COMSOL on your desktoo or laptop computer.
#* You will require access to the COMSOL license server running on Carbon, either through VPN (fairly easy) or SSH tunneling (can be intimidating for first-time users). See below for details.

=== Running a batch calculation ===
When the computational stage of models takes longer than a few minutes to complete, it will be beneficial to submit save the model file and run it in parallel on Carbon.
See the [[#Batch jobs|section below]].

== Use with X11 ==
For casual use, and with [[HPC/Network Access/SSH Tunnel Setup on Linux and MacOS | X11 tunneling to Carbon]] set up, simply add to your <code>~/.bashrc</code> file on Carbon:
module load comsol

Then, from the command line on a Carbon login node, start the COMSOL GUI:
comsol &

License configuration is automatic on Carbon.

Comsol will use a number of cores to run calculations multithreaded by default. (On Carbon, COMSOL_NUM_THREADS is set.)

'''Quit the application''' as soon as you are done. Only one user can use Comsol at a time.

== Use with VNC ==
See general [[HPC/Network Access/Virtual Desktop | instructions on using VNC]], and the COMSOL example therein.

== Native desktop version ==

To get the fastest response from the COMSOL GUI, run a native COMSOL version on your desktop machine or laptop, called ''client machine'' below.

Enter license details either in the COMSOL Setup configuration dialog.

== Native desktop version — License configuration ==
A client's COMSOL license needs to be configured depending on how the client reaches CNM network resources. Follow the appropriate section:
* [[#License configuration when inside CNM or using VPN]]
* [[#License configuration when outside CNM and using SSH Tunneling]]

=== License configuration when inside CNM or using VPN ===
==== Ensure correct DNS search domains ====

* Inspect the client machine's network configuration.
* Ensure that '''two''' entries are listed as DNS Search Domains for the current network profile:
nst.anl.gov
cnm.anl.gov
: (The order of entries does not matter for the purposes considered here.)
* If the client machine is ''wired to CNM networks'', the domains are normally provided correctly by DHCP.
* If the client machine is ''connected over VPN'', you may need edit the respective network profile.
:* On a Mac, find the VPN configuration details under System Preferences, Network, Advanced… . You may need to unlock the Network panel first before you can make changes under "Advanced…". It appears that those changes are being saved and will be activated automatically the next time that this VPN session gets started.
: [[Image:VNC_add_cnm_to_DNS_1.png|400px]] [[Image:VNC_add_cnm_to_DNS_2.png|400px]]
:* On Windows or Linux machines, take ''analogous'' steps, using the platform's own configuration tools.

* Exclude license manager components from client desktop installation. (Those are all handled server-side.)

==== License format ====
* Floating Network License (FNL), "Three-server redundancy"

==== License servers ====
* On client machines, while running the COMSOL installer or license configuration, specify license servers and ports as follows:
clicense1 1718
clicense2 1718
clicense3 1718
: '''Note:''' Use ''short host name'' form exactly as given here. (Technical Reason: The license must be usable both on Carbon's internal network as well as on divisional networks that are outside of Carbon).

'''Advanced:''' A client machine's COMSOL license file <code><COMSOLDIR>/Multiphysics/license/license.dat</code> should look as follows:
<source lang="make">
SERVER clicense1 ANY 1718
SERVER clicense2 ANY 1718
SERVER clicense3 ANY 1718
USE_SERVER
</source>

<code><COMSOLDIR></code> is located by default at:
* '''Windows''': C:\Program files\COMSOL\COMSOL''nn''
* '''Linux''': /usr/local/comsol''nn''
* '''Mac''': /Applications/COMSOL''nn''
("''nn''" is the application's abbreviated version number.)

==== Network Troubleshooting ====
When using the license access model descibed here (as opposed to tunneled over SSH), ensure the following:
* Verify that short host names can be resolved to IP addresses. Either one of the following commands should produce sensible output:
host clicense1
: or:
ping -c 3 clicense1
* Verify that ''both license ports on the same server'' can be reached from your client machine:
telnet '''clicense1''' 1718
telnet '''clicense1''' 1719
: or:
telnet '''clicense2''' 1718
telnet '''clicense2''' 1719
: A ''successful'' connection looks like:
Trying 146.139.x.y...
Connected to clicense1.cnm.anl.gov.
Escape character is '^]'.
:: Do not enter any data. Instead, close each connection by hitting <code>Ctrl</code><code>]</code> (closing square bracket), followed by <code>Ctrl</code><code>D</code>.
: A ''failed'' connection gives one of these errors:
telnet: connect to address 146.139.x.y: Operation timed out
: or:
telnet: connect to address 146.139.x.y: Connection refused
:: In these cases, ask that your IP firewall configuration be reviewed and changed.

=== License configuration when outside CNM and using SSH Tunneling ===
If you're comfortable with ssh and tunneling, configure client licensing as follows:

==== License format ====
* Floating Network License (FNL), "Port number and Hostname"
: '''Note:''' Three-server redundancy cannot be accessed over SSH tunneling. Instead, the user must configure just one server, and if needed in rare situations manually reconfigure to use the other server if the first one fails.

==== License server ====
* On running the COMSOL installer, provide as license source:
** Server: 127.0.0.1
** Port: 1718



Connect to <code>mega</code> or a Carbon login node with one of the following ssh commands:
* First, set up a tunnel to the primary license server:
ssh -L 1718:'''clicense1''':1718 -L 1719:'''clicense1''':1719 mega.cnm.anl.gov
* Start COMSOL.
: If you get an error that the number of licenses is exhausted, that is at least confirmation that you correctly reached the license server.

If the above attempt takes a long time or hangs entirely, it may be because the license service currently runs on the secondary server.
* Connect to the alternate license server:
ssh -L 1718:'''clicense2''':1718 -L 1719:'''clicense2''':1719 mega.cnm.anl.gov
* Start COMSOL.

== License Troubleshooting ==
The following are typical error messages.
=== Could not obtain license for COMSOL Multiphysics GUI ===
* Error details:
License error: -4.
'''Licensed number of users already reached.'''
Feature: COMSOLGUI
License path: /Users/stern/Applications/COMSOL52a/license/license.dat:
FlexNet Licensing error:-4,132
* Reason: The connection to the license server succeeded, but the license token(s) you requested are in use.
* Solution: Wait a bit, or [http://carbonmgmt/ganglia/lic.php identify who's using COMSOL] and politely ask the user when they might be finished with their work.

=== License server machine is down or not responding ===
* Error details:
License error: -96.
See the system administrator about starting the license server system, or
make sure you're referring to the right host (see LM_LICENSE_FILE).
Feature: SERIAL
Hostname: clicense3
License path: /Users/stern/Applications/COMSOL52a/license/license.dat:
FlexNet Licensing error:-96,7
* Reason: The license servers cannot be reached at all. The <code>ping</code> or <code>host</code> tests mentioned above fail.
* Solution: Review the client license file and DNS lookup domains.

=== License file does not support this version ===
* Error details:
Feature: SERIAL
'''Application version > License version: 5.21 > 3.5'''
License path: /Users/stern/Applications/COMSOL52a/license/license.dat:
FlexNet Licensing error:-21,126
* Reason: Only the first of two steps for connecting to the license servers succeeds, which may happen when the redundant license service runs on the alternate server instead of the one you're connected to.
* Solution: When using SSH, try <code>clicense2</code> instead of <code>clicense1</code>, or vice versa. Revisit your <code>~/.ssh/config</code> file or use the alternate command line as shown in the SSH section above.

== Multiple CPU cores ==
Ways to start multi-core (from http://www.comsol.com/support/knowledgebase/1096/)
comsol -np 8

export COMSOL_NUM_THREADS=8 # automatically provided on ''Carbon''
comsol
* COMSOL does not benefit from hyperthreading (oversubscribing the physical core count).

== Batch jobs ==

* http://www.comsol.com/support/knowledgebase/1001/ .
* [[Media:HPC 2012-05-16 Comsol parallel usage.pdf]] (Modified from http://www.comsol.com/support/knowledgebase/1001/files/4027/cluster_install_linux_42a.pptx – Appendix slide, PBS)
* '''Note:''' MPD is not supported on Carbon (I patched the main comsol script), and no longer necessary at all in Comsol-4.3.

=== Sample Job file ===
See:
$COMSOL_HOME/sample.job

== Important COMSOL Support Articles ==
* Toplevel suport page: https://www.comsol.com/support

=== KB Articles ===
* [https://www.comsol.com/support/knowledgebase/1001 Running COMSOL® in parallel on clusters]
* [https://www.comsol.com/support/knowledgebase/1250 Running parametric sweeps, batch sweeps, and cluster sweeps from the command line]
* [https://www.comsol.com/support/knowledgebase/1255 Reducing the amount of solution data stored in a model]
* [https://www.comsol.com/support/knowledgebase/830 Handling large models]

=== Blog-style articles ===
* [https://www.comsol.com/blogs/the-power-of-the-batch-sweep/ The Power of the Batch Sweep]
* [https://www.comsol.com/blogs/how-to-run-simulations-in-batch-mode-from-the-command-line/ How to Run Simulations in Batch Mode from the Command Line]

HPC/Applications/comsol

2025-08-07T15:37:01Z

Stern: /* COMSOL Multiphysics */

== Introduction ==
The [http://www.comsol.com/products/multiphysics/ '''COMSOL Multiphysics'''] application suite
is available for Mac, Windows, and Linux.

== Access ==
[[Image:HPC Comsol licensed modules.png|right|450px|thumb|COMSOL features licensed at CNM as of 2024-10.]]
=== COMSOL Multiphysics ===
Access to '''COMSOL Multiphysics''', the main application with both a graphical user interface and batch computing capabilities, will be granted '''''upon request''''' .

You must be:
* An '''''Argonne employee'''''  in one of the following roles:
** '''''NST staff member'''''  (including PostDocs, joint appointments, and students at NST).
** '''''CNM Facility User'''''  (including ''non-NST Argonne employees''), with an active proposal that has COMSOL in scope. Please see [[HPC/FAQ#Access_and_proposals]].

The set of available COMSOL modules that we have licensed is shown at right.

* The native desktop version can be installed only on ''Argonne-owned'' computers (laptop or desktop). Please contact [mailto:help@anl.gov?Subject=COMSOL%20license%20access%20request CNM support]. Make sure to review [http://www.comsol.com/support/updates/ Product Updates].
* To run COMSOL on our HPC system ''Carbon'', as GUI or as batch computing jobs, your proposal must have ''Carbon'' access in scope.

=== COMSOL Server ===
''CNM Facility Users'' who are not Argonne employees are not eligible to access
the full '''COMSOL Multiphysics''' product, in compliance with its licensing terms and conditions.
However, the [https://www.comsol.com/comsol-server '''COMSOL Server'''] product offers an alternative
albeit more narrow means of access.

All users, including '''''non-Argonne users''''' may be able to run, over a web interface in their browser,
pre-parameterized "fixed-model" simulations that were made with the '''COMSOL Application Builder''' (requires Windows).

To inquire about using COMSOL server applications,
please contact us.

We will review possible purchase paths for COMSOL Server, and the ''considerable effort'' likely required by users and the CNM to develop and deploy a COMSOL application.

== Support ==
* For questions on how to run or start COMSOL:
*: Submit an [[HPC/Support | '''IT support request''']]
* For questions related to most COMSOL data or scripts, contact COMSOL support at:
*: [mailto:support@comsol.com '''support'''@comsol.com]
* '''Exception:''' If your support question might contain data that is subject to '''US Export Control''' regulations, contact instead:
*: [mailto:us-export-control@us.comsol.com '''us-export-control'''@'''us'''.comsol.com]

== The modes to run COMSOL ==
There are several ways of running COMSOL.

=== Running the desktop application ===
To edit models and analyze results, use the COMSOL graphical user interface (GUI). You can run it in several ways, each explained in more detail in the following sections.

# Run on Carbon, display on your desktop using the [http://en.wikipedia.org/wiki/X_Window_System X Window system (X11)].
#* Requires an ''X11 Server'' application running on your client machine, and SSH X11 forwarding.
#* The X11 protocol can be slow over LAN.
# Run on Carbon within a [[HPC/Network Access/Virtual Desktop | virtual desktop environment (VNC)]].
#* Requires a VNC client on your machine.
#* It can be confusing where VNC and the app runs.
#* The virtual desktop presents a (deliberatly) sparse windowing environment – Copy&Paste can be difficult to use.
# Run on your own computer, remotely checking out Carbon's license.
#* Requires a full installation of COMSOL on your desktoo or laptop computer.
#* You will require access to the COMSOL license server running on Carbon, either through VPN (fairly easy) or SSH tunneling (can be intimidating for first-time users). See below for details.

=== Running a batch calculation ===
When the computational stage of models takes longer than a few minutes to complete, it will be beneficial to submit save the model file and run it in parallel on Carbon.
See the [[#Batch jobs|section below]].

== Use with X11 ==
For casual use, and with [[HPC/Network Access/SSH Tunnel Setup on Linux and MacOS | X11 tunneling to Carbon]] set up, simply add to your <code>~/.bashrc</code> file on Carbon:
module load comsol

Then, from the command line on a Carbon login node, start the COMSOL GUI:
comsol &

License configuration is automatic on Carbon.

Comsol will use a number of cores to run calculations multithreaded by default. (On Carbon, COMSOL_NUM_THREADS is set.)

'''Quit the application''' as soon as you are done. Only one user can use Comsol at a time.

== Use with VNC ==
See general [[HPC/Network Access/Virtual Desktop | instructions on using VNC]], and the COMSOL example therein.

== Native desktop version ==

To get the fastest response from the COMSOL GUI, run a native COMSOL version on your desktop machine or laptop, called ''client machine'' below.

Enter license details either in the COMSOL Setup configuration dialog.

== Native desktop version — License configuration ==
A client's COMSOL license needs to be configured depending on how the client reaches CNM network resources. Follow the appropriate section:
* [[#License configuration when inside CNM or using VPN]]
* [[#License configuration when outside CNM and using SSH Tunneling]]

=== License configuration when inside CNM or using VPN ===
==== Ensure correct DNS search domains ====

* Inspect the client machine's network configuration.
* Ensure that '''two''' entries are listed as DNS Search Domains for the current network profile:
nst.anl.gov
cnm.anl.gov
: (The order of entries does not matter for the purposes considered here.)
* If the client machine is ''wired to CNM networks'', the domains are normally provided correctly by DHCP.
* If the client machine is ''connected over VPN'', you may need edit the respective network profile.
:* On a Mac, find the VPN configuration details under System Preferences, Network, Advanced… . You may need to unlock the Network panel first before you can make changes under "Advanced…". It appears that those changes are being saved and will be activated automatically the next time that this VPN session gets started.
: [[Image:VNC_add_cnm_to_DNS_1.png|400px]] [[Image:VNC_add_cnm_to_DNS_2.png|400px]]
:* On Windows or Linux machines, take ''analogous'' steps, using the platform's own configuration tools.

* Exclude license manager components from client desktop installation. (Those are all handled server-side.)

==== License format ====
* Floating Network License (FNL), "Three-server redundancy"

==== License servers ====
* On client machines, while running the COMSOL installer or license configuration, specify license servers and ports as follows:
clicense1 1718
clicense2 1718
clicense3 1718
: '''Note:''' Use ''short host name'' form exactly as given here. (Technical Reason: The license must be usable both on Carbon's internal network as well as on divisional networks that are outside of Carbon).

'''Advanced:''' A client machine's COMSOL license file <code><COMSOLDIR>/Multiphysics/license/license.dat</code> should look as follows:
<source lang="make">
SERVER clicense1 ANY 1718
SERVER clicense2 ANY 1718
SERVER clicense3 ANY 1718
USE_SERVER
</source>

<code><COMSOLDIR></code> is located by default at:
* '''Windows''': C:\Program files\COMSOL\COMSOL''nn''
* '''Linux''': /usr/local/comsol''nn''
* '''Mac''': /Applications/COMSOL''nn''
("''nn''" is the application's abbreviated version number.)

==== Network Troubleshooting ====
When using the license access model descibed here (as opposed to tunneled over SSH), ensure the following:
* Verify that short host names can be resolved to IP addresses. Either one of the following commands should produce sensible output:
host clicense1
: or:
ping -c 3 clicense1
* Verify that ''both license ports on the same server'' can be reached from your client machine:
telnet '''clicense1''' 1718
telnet '''clicense1''' 1719
: or:
telnet '''clicense2''' 1718
telnet '''clicense2''' 1719
: A ''successful'' connection looks like:
Trying 146.139.x.y...
Connected to clicense1.cnm.anl.gov.
Escape character is '^]'.
:: Do not enter any data. Instead, close each connection by hitting <code>Ctrl</code><code>]</code> (closing square bracket), followed by <code>Ctrl</code><code>D</code>.
: A ''failed'' connection gives one of these errors:
telnet: connect to address 146.139.x.y: Operation timed out
: or:
telnet: connect to address 146.139.x.y: Connection refused
:: In these cases, ask that your IP firewall configuration be reviewed and changed.

=== License configuration when outside CNM and using SSH Tunneling ===
If you're comfortable with ssh and tunneling, configure client licensing as follows:

==== License format ====
* Floating Network License (FNL), "Port number and Hostname"
: '''Note:''' Three-server redundancy cannot be accessed over SSH tunneling. Instead, the user must configure just one server, and if needed in rare situations manually reconfigure to use the other server if the first one fails.

==== License server ====
* Enter into the COMSOL license file:
<source lang="make">
SERVER 127.0.0.1 ANY 1718
USE_SERVER
</source>
: '''Note:''' Do not use "localhost", as this will be directly (and incorrectly) interpreted by COMSOL to mean that the current machine actually ''is'' a license server.

Connect to <code>mega</code> or a Carbon login node with one of the following ssh commands:
* First, set up a tunnel to the primary license server:
ssh -L 1718:'''clicense1''':1718 -L 1719:'''clicense1''':1719 mega
* Start COMSOL.
: If you get an error that the number of licenses is exhausted, that is at least confirmation that you correctly reached the license server.

If the above attempt takes a long time or hangs entirely, it may be because the license service currently runs on the secondary server.
* Connect to the alternate license server:
ssh -L 1718:'''clicense2''':1718 -L 1719:'''clicense2''':1719 mega
* Start COMSOL.

== License Troubleshooting ==
The following are typical error messages.
=== Could not obtain license for COMSOL Multiphysics GUI ===
* Error details:
License error: -4.
'''Licensed number of users already reached.'''
Feature: COMSOLGUI
License path: /Users/stern/Applications/COMSOL52a/license/license.dat:
FlexNet Licensing error:-4,132
* Reason: The connection to the license server succeeded, but the license token(s) you requested are in use.
* Solution: Wait a bit, or [http://carbonmgmt/ganglia/lic.php identify who's using COMSOL] and politely ask the user when they might be finished with their work.

=== License server machine is down or not responding ===
* Error details:
License error: -96.
See the system administrator about starting the license server system, or
make sure you're referring to the right host (see LM_LICENSE_FILE).
Feature: SERIAL
Hostname: clicense3
License path: /Users/stern/Applications/COMSOL52a/license/license.dat:
FlexNet Licensing error:-96,7
* Reason: The license servers cannot be reached at all. The <code>ping</code> or <code>host</code> tests mentioned above fail.
* Solution: Review the client license file and DNS lookup domains.

=== License file does not support this version ===
* Error details:
Feature: SERIAL
'''Application version > License version: 5.21 > 3.5'''
License path: /Users/stern/Applications/COMSOL52a/license/license.dat:
FlexNet Licensing error:-21,126
* Reason: Only the first of two steps for connecting to the license servers succeeds, which may happen when the redundant license service runs on the alternate server instead of the one you're connected to.
* Solution: When using SSH, try <code>clicense2</code> instead of <code>clicense1</code>, or vice versa. Revisit your <code>~/.ssh/config</code> file or use the alternate command line as shown in the SSH section above.

== Multiple CPU cores ==
Ways to start multi-core (from http://www.comsol.com/support/knowledgebase/1096/)
comsol -np 8

export COMSOL_NUM_THREADS=8 # automatically provided on ''Carbon''
comsol
* COMSOL does not benefit from hyperthreading (oversubscribing the physical core count).

== Batch jobs ==

* http://www.comsol.com/support/knowledgebase/1001/ .
* [[Media:HPC 2012-05-16 Comsol parallel usage.pdf]] (Modified from http://www.comsol.com/support/knowledgebase/1001/files/4027/cluster_install_linux_42a.pptx – Appendix slide, PBS)
* '''Note:''' MPD is not supported on Carbon (I patched the main comsol script), and no longer necessary at all in Comsol-4.3.

=== Sample Job file ===
See:
$COMSOL_HOME/sample.job

== Important COMSOL Support Articles ==
* Toplevel suport page: https://www.comsol.com/support

=== KB Articles ===
* [https://www.comsol.com/support/knowledgebase/1001 Running COMSOL® in parallel on clusters]
* [https://www.comsol.com/support/knowledgebase/1250 Running parametric sweeps, batch sweeps, and cluster sweeps from the command line]
* [https://www.comsol.com/support/knowledgebase/1255 Reducing the amount of solution data stored in a model]
* [https://www.comsol.com/support/knowledgebase/830 Handling large models]

=== Blog-style articles ===
* [https://www.comsol.com/blogs/the-power-of-the-batch-sweep/ The Power of the Batch Sweep]
* [https://www.comsol.com/blogs/how-to-run-simulations-in-batch-mode-from-the-command-line/ How to Run Simulations in Batch Mode from the Command Line]

HPC/Applications/comsol

2025-08-07T15:33:57Z

Stern: /* COMSOL Multiphysics */

== Introduction ==
The [http://www.comsol.com/products/multiphysics/ '''COMSOL Multiphysics'''] application suite
is available for Mac, Windows, and Linux.

== Access ==
[[Image:HPC Comsol licensed modules.png|right|450px|thumb|COMSOL features licensed at CNM as of 2024-10.]]
=== COMSOL Multiphysics ===
Access to '''COMSOL Multiphysics''', the main application with both a graphical user interface and batch computing capabilities, will be granted '''''upon request''''' .

You must be:
* An '''''Argonne employee'''''  in one of the following roles:
** '''''NST staff member'''''  (including PostDocs, joint appointments, and students at NST).
** '''''CNM Facility User'''''  (including ''non-NST Argonne employees''), with an active proposal that has COMSOL in scope. Please see [[HPC/FAQ#Access_and_proposals]].

The set of available COMSOL modules that we have licensed is shown at right.

* To have a native desktop version installed on your Argonne-owned computer (laptop or desktop), please contact [mailto:help@anl.gov?Subject=COMSOL%20license%20access%20request CNM support]. Make sure to review [http://www.comsol.com/support/updates/ Product Updates].
* To run COMSOL on our HPC system ''Carbon'', as GUI or as batch computing jobs, your proposal must have ''Carbon'' access in scope.

=== COMSOL Server ===
''CNM Facility Users'' who are not Argonne employees are not eligible to access
the full '''COMSOL Multiphysics''' product, in compliance with its licensing terms and conditions.
However, the [https://www.comsol.com/comsol-server '''COMSOL Server'''] product offers an alternative
albeit more narrow means of access.

All users, including '''''non-Argonne users''''' may be able to run, over a web interface in their browser,
pre-parameterized "fixed-model" simulations that were made with the '''COMSOL Application Builder''' (requires Windows).

To inquire about using COMSOL server applications,
please contact us.

We will review possible purchase paths for COMSOL Server, and the ''considerable effort'' likely required by users and the CNM to develop and deploy a COMSOL application.

== Support ==
* For questions on how to run or start COMSOL:
*: Submit an [[HPC/Support | '''IT support request''']]
* For questions related to most COMSOL data or scripts, contact COMSOL support at:
*: [mailto:support@comsol.com '''support'''@comsol.com]
* '''Exception:''' If your support question might contain data that is subject to '''US Export Control''' regulations, contact instead:
*: [mailto:us-export-control@us.comsol.com '''us-export-control'''@'''us'''.comsol.com]

== The modes to run COMSOL ==
There are several ways of running COMSOL.

=== Running the desktop application ===
To edit models and analyze results, use the COMSOL graphical user interface (GUI). You can run it in several ways, each explained in more detail in the following sections.

# Run on Carbon, display on your desktop using the [http://en.wikipedia.org/wiki/X_Window_System X Window system (X11)].
#* Requires an ''X11 Server'' application running on your client machine, and SSH X11 forwarding.
#* The X11 protocol can be slow over LAN.
# Run on Carbon within a [[HPC/Network Access/Virtual Desktop | virtual desktop environment (VNC)]].
#* Requires a VNC client on your machine.
#* It can be confusing where VNC and the app runs.
#* The virtual desktop presents a (deliberatly) sparse windowing environment – Copy&Paste can be difficult to use.
# Run on your own computer, remotely checking out Carbon's license.
#* Requires a full installation of COMSOL on your desktoo or laptop computer.
#* You will require access to the COMSOL license server running on Carbon, either through VPN (fairly easy) or SSH tunneling (can be intimidating for first-time users). See below for details.

=== Running a batch calculation ===
When the computational stage of models takes longer than a few minutes to complete, it will be beneficial to submit save the model file and run it in parallel on Carbon.
See the [[#Batch jobs|section below]].

== Use with X11 ==
For casual use, and with [[HPC/Network Access/SSH Tunnel Setup on Linux and MacOS | X11 tunneling to Carbon]] set up, simply add to your <code>~/.bashrc</code> file on Carbon:
module load comsol

Then, from the command line on a Carbon login node, start the COMSOL GUI:
comsol &

License configuration is automatic on Carbon.

Comsol will use a number of cores to run calculations multithreaded by default. (On Carbon, COMSOL_NUM_THREADS is set.)

'''Quit the application''' as soon as you are done. Only one user can use Comsol at a time.

== Use with VNC ==
See general [[HPC/Network Access/Virtual Desktop | instructions on using VNC]], and the COMSOL example therein.

== Native desktop version ==

To get the fastest response from the COMSOL GUI, run a native COMSOL version on your desktop machine or laptop, called ''client machine'' below.

Enter license details either in the COMSOL Setup configuration dialog.

== Native desktop version — License configuration ==
A client's COMSOL license needs to be configured depending on how the client reaches CNM network resources. Follow the appropriate section:
* [[#License configuration when inside CNM or using VPN]]
* [[#License configuration when outside CNM and using SSH Tunneling]]

=== License configuration when inside CNM or using VPN ===
==== Ensure correct DNS search domains ====

* Inspect the client machine's network configuration.
* Ensure that '''two''' entries are listed as DNS Search Domains for the current network profile:
nst.anl.gov
cnm.anl.gov
: (The order of entries does not matter for the purposes considered here.)
* If the client machine is ''wired to CNM networks'', the domains are normally provided correctly by DHCP.
* If the client machine is ''connected over VPN'', you may need edit the respective network profile.
:* On a Mac, find the VPN configuration details under System Preferences, Network, Advanced… . You may need to unlock the Network panel first before you can make changes under "Advanced…". It appears that those changes are being saved and will be activated automatically the next time that this VPN session gets started.
: [[Image:VNC_add_cnm_to_DNS_1.png|400px]] [[Image:VNC_add_cnm_to_DNS_2.png|400px]]
:* On Windows or Linux machines, take ''analogous'' steps, using the platform's own configuration tools.

* Exclude license manager components from client desktop installation. (Those are all handled server-side.)

==== License format ====
* Floating Network License (FNL), "Three-server redundancy"

==== License servers ====
* On client machines, while running the COMSOL installer or license configuration, specify license servers and ports as follows:
clicense1 1718
clicense2 1718
clicense3 1718
: '''Note:''' Use ''short host name'' form exactly as given here. (Technical Reason: The license must be usable both on Carbon's internal network as well as on divisional networks that are outside of Carbon).

'''Advanced:''' A client machine's COMSOL license file <code><COMSOLDIR>/Multiphysics/license/license.dat</code> should look as follows:
<source lang="make">
SERVER clicense1 ANY 1718
SERVER clicense2 ANY 1718
SERVER clicense3 ANY 1718
USE_SERVER
</source>

<code><COMSOLDIR></code> is located by default at:
* '''Windows''': C:\Program files\COMSOL\COMSOL''nn''
* '''Linux''': /usr/local/comsol''nn''
* '''Mac''': /Applications/COMSOL''nn''
("''nn''" is the application's abbreviated version number.)

==== Network Troubleshooting ====
When using the license access model descibed here (as opposed to tunneled over SSH), ensure the following:
* Verify that short host names can be resolved to IP addresses. Either one of the following commands should produce sensible output:
host clicense1
: or:
ping -c 3 clicense1
* Verify that ''both license ports on the same server'' can be reached from your client machine:
telnet '''clicense1''' 1718
telnet '''clicense1''' 1719
: or:
telnet '''clicense2''' 1718
telnet '''clicense2''' 1719
: A ''successful'' connection looks like:
Trying 146.139.x.y...
Connected to clicense1.cnm.anl.gov.
Escape character is '^]'.
:: Do not enter any data. Instead, close each connection by hitting <code>Ctrl</code><code>]</code> (closing square bracket), followed by <code>Ctrl</code><code>D</code>.
: A ''failed'' connection gives one of these errors:
telnet: connect to address 146.139.x.y: Operation timed out
: or:
telnet: connect to address 146.139.x.y: Connection refused
:: In these cases, ask that your IP firewall configuration be reviewed and changed.

=== License configuration when outside CNM and using SSH Tunneling ===
If you're comfortable with ssh and tunneling, configure client licensing as follows:

==== License format ====
* Floating Network License (FNL), "Port number and Hostname"
: '''Note:''' Three-server redundancy cannot be accessed over SSH tunneling. Instead, the user must configure just one server, and if needed in rare situations manually reconfigure to use the other server if the first one fails.

==== License server ====
* Enter into the COMSOL license file:
<source lang="make">
SERVER 127.0.0.1 ANY 1718
USE_SERVER
</source>
: '''Note:''' Do not use "localhost", as this will be directly (and incorrectly) interpreted by COMSOL to mean that the current machine actually ''is'' a license server.

Connect to <code>mega</code> or a Carbon login node with one of the following ssh commands:
* First, set up a tunnel to the primary license server:
ssh -L 1718:'''clicense1''':1718 -L 1719:'''clicense1''':1719 mega
* Start COMSOL.
: If you get an error that the number of licenses is exhausted, that is at least confirmation that you correctly reached the license server.

If the above attempt takes a long time or hangs entirely, it may be because the license service currently runs on the secondary server.
* Connect to the alternate license server:
ssh -L 1718:'''clicense2''':1718 -L 1719:'''clicense2''':1719 mega
* Start COMSOL.

== License Troubleshooting ==
The following are typical error messages.
=== Could not obtain license for COMSOL Multiphysics GUI ===
* Error details:
License error: -4.
'''Licensed number of users already reached.'''
Feature: COMSOLGUI
License path: /Users/stern/Applications/COMSOL52a/license/license.dat:
FlexNet Licensing error:-4,132
* Reason: The connection to the license server succeeded, but the license token(s) you requested are in use.
* Solution: Wait a bit, or [http://carbonmgmt/ganglia/lic.php identify who's using COMSOL] and politely ask the user when they might be finished with their work.

=== License server machine is down or not responding ===
* Error details:
License error: -96.
See the system administrator about starting the license server system, or
make sure you're referring to the right host (see LM_LICENSE_FILE).
Feature: SERIAL
Hostname: clicense3
License path: /Users/stern/Applications/COMSOL52a/license/license.dat:
FlexNet Licensing error:-96,7
* Reason: The license servers cannot be reached at all. The <code>ping</code> or <code>host</code> tests mentioned above fail.
* Solution: Review the client license file and DNS lookup domains.

=== License file does not support this version ===
* Error details:
Feature: SERIAL
'''Application version > License version: 5.21 > 3.5'''
License path: /Users/stern/Applications/COMSOL52a/license/license.dat:
FlexNet Licensing error:-21,126
* Reason: Only the first of two steps for connecting to the license servers succeeds, which may happen when the redundant license service runs on the alternate server instead of the one you're connected to.
* Solution: When using SSH, try <code>clicense2</code> instead of <code>clicense1</code>, or vice versa. Revisit your <code>~/.ssh/config</code> file or use the alternate command line as shown in the SSH section above.

== Multiple CPU cores ==
Ways to start multi-core (from http://www.comsol.com/support/knowledgebase/1096/)
comsol -np 8

export COMSOL_NUM_THREADS=8 # automatically provided on ''Carbon''
comsol
* COMSOL does not benefit from hyperthreading (oversubscribing the physical core count).

== Batch jobs ==

* http://www.comsol.com/support/knowledgebase/1001/ .
* [[Media:HPC 2012-05-16 Comsol parallel usage.pdf]] (Modified from http://www.comsol.com/support/knowledgebase/1001/files/4027/cluster_install_linux_42a.pptx – Appendix slide, PBS)
* '''Note:''' MPD is not supported on Carbon (I patched the main comsol script), and no longer necessary at all in Comsol-4.3.

=== Sample Job file ===
See:
$COMSOL_HOME/sample.job

== Important COMSOL Support Articles ==
* Toplevel suport page: https://www.comsol.com/support

=== KB Articles ===
* [https://www.comsol.com/support/knowledgebase/1001 Running COMSOL® in parallel on clusters]
* [https://www.comsol.com/support/knowledgebase/1250 Running parametric sweeps, batch sweeps, and cluster sweeps from the command line]
* [https://www.comsol.com/support/knowledgebase/1255 Reducing the amount of solution data stored in a model]
* [https://www.comsol.com/support/knowledgebase/830 Handling large models]

=== Blog-style articles ===
* [https://www.comsol.com/blogs/the-power-of-the-batch-sweep/ The Power of the Batch Sweep]
* [https://www.comsol.com/blogs/how-to-run-simulations-in-batch-mode-from-the-command-line/ How to Run Simulations in Batch Mode from the Command Line]

HPC/Applications/comsol

2025-08-07T15:32:31Z

Stern: /* COMSOL Multiphysics */

== Introduction ==
The [http://www.comsol.com/products/multiphysics/ '''COMSOL Multiphysics'''] application suite
is available for Mac, Windows, and Linux.

== Access ==
[[Image:HPC Comsol licensed modules.png|right|450px|thumb|COMSOL features licensed at CNM as of 2024-10.]]
=== COMSOL Multiphysics ===
Access to '''COMSOL Multiphysics''', the main application with both a graphical user interface and batch computing capabilities, will be granted '''''upon request''''' .

You must be:
* An '''''Argonne employee'''''  in one of the following roles:
** '''''NST staff member'''''  (including PostDocs, joint appointments, and students at NST).
** '''''CNM Facility User'''''  (including ''non-NST Argonne employees''), with an active proposal that has COMSOL in scope. This includes . — please see [[HPC/FAQ#Access_and_proposals]].

The set of available COMSOL modules that we have licensed is shown at right.

* To have a native desktop version installed on your Argonne-owned computer (laptop or desktop), please contact [mailto:help@anl.gov?Subject=COMSOL%20license%20access%20request CNM support]. Make sure to review [http://www.comsol.com/support/updates/ Product Updates].
* To run COMSOL on our HPC system ''Carbon'', as GUI or as batch computing jobs, your proposal must have ''Carbon'' access in scope.

=== COMSOL Server ===
''CNM Facility Users'' who are not Argonne employees are not eligible to access
the full '''COMSOL Multiphysics''' product, in compliance with its licensing terms and conditions.
However, the [https://www.comsol.com/comsol-server '''COMSOL Server'''] product offers an alternative
albeit more narrow means of access.

All users, including '''''non-Argonne users''''' may be able to run, over a web interface in their browser,
pre-parameterized "fixed-model" simulations that were made with the '''COMSOL Application Builder''' (requires Windows).

To inquire about using COMSOL server applications,
please contact us.

We will review possible purchase paths for COMSOL Server, and the ''considerable effort'' likely required by users and the CNM to develop and deploy a COMSOL application.

== Support ==
* For questions on how to run or start COMSOL:
*: Submit an [[HPC/Support | '''IT support request''']]
* For questions related to most COMSOL data or scripts, contact COMSOL support at:
*: [mailto:support@comsol.com '''support'''@comsol.com]
* '''Exception:''' If your support question might contain data that is subject to '''US Export Control''' regulations, contact instead:
*: [mailto:us-export-control@us.comsol.com '''us-export-control'''@'''us'''.comsol.com]

== The modes to run COMSOL ==
There are several ways of running COMSOL.

=== Running the desktop application ===
To edit models and analyze results, use the COMSOL graphical user interface (GUI). You can run it in several ways, each explained in more detail in the following sections.

# Run on Carbon, display on your desktop using the [http://en.wikipedia.org/wiki/X_Window_System X Window system (X11)].
#* Requires an ''X11 Server'' application running on your client machine, and SSH X11 forwarding.
#* The X11 protocol can be slow over LAN.
# Run on Carbon within a [[HPC/Network Access/Virtual Desktop | virtual desktop environment (VNC)]].
#* Requires a VNC client on your machine.
#* It can be confusing where VNC and the app runs.
#* The virtual desktop presents a (deliberatly) sparse windowing environment – Copy&Paste can be difficult to use.
# Run on your own computer, remotely checking out Carbon's license.
#* Requires a full installation of COMSOL on your desktoo or laptop computer.
#* You will require access to the COMSOL license server running on Carbon, either through VPN (fairly easy) or SSH tunneling (can be intimidating for first-time users). See below for details.

=== Running a batch calculation ===
When the computational stage of models takes longer than a few minutes to complete, it will be beneficial to submit save the model file and run it in parallel on Carbon.
See the [[#Batch jobs|section below]].

== Use with X11 ==
For casual use, and with [[HPC/Network Access/SSH Tunnel Setup on Linux and MacOS | X11 tunneling to Carbon]] set up, simply add to your <code>~/.bashrc</code> file on Carbon:
module load comsol

Then, from the command line on a Carbon login node, start the COMSOL GUI:
comsol &

License configuration is automatic on Carbon.

Comsol will use a number of cores to run calculations multithreaded by default. (On Carbon, COMSOL_NUM_THREADS is set.)

'''Quit the application''' as soon as you are done. Only one user can use Comsol at a time.

== Use with VNC ==
See general [[HPC/Network Access/Virtual Desktop | instructions on using VNC]], and the COMSOL example therein.

== Native desktop version ==

To get the fastest response from the COMSOL GUI, run a native COMSOL version on your desktop machine or laptop, called ''client machine'' below.

Enter license details either in the COMSOL Setup configuration dialog.

== Native desktop version — License configuration ==
A client's COMSOL license needs to be configured depending on how the client reaches CNM network resources. Follow the appropriate section:
* [[#License configuration when inside CNM or using VPN]]
* [[#License configuration when outside CNM and using SSH Tunneling]]

=== License configuration when inside CNM or using VPN ===
==== Ensure correct DNS search domains ====

* Inspect the client machine's network configuration.
* Ensure that '''two''' entries are listed as DNS Search Domains for the current network profile:
nst.anl.gov
cnm.anl.gov
: (The order of entries does not matter for the purposes considered here.)
* If the client machine is ''wired to CNM networks'', the domains are normally provided correctly by DHCP.
* If the client machine is ''connected over VPN'', you may need edit the respective network profile.
:* On a Mac, find the VPN configuration details under System Preferences, Network, Advanced… . You may need to unlock the Network panel first before you can make changes under "Advanced…". It appears that those changes are being saved and will be activated automatically the next time that this VPN session gets started.
: [[Image:VNC_add_cnm_to_DNS_1.png|400px]] [[Image:VNC_add_cnm_to_DNS_2.png|400px]]
:* On Windows or Linux machines, take ''analogous'' steps, using the platform's own configuration tools.

* Exclude license manager components from client desktop installation. (Those are all handled server-side.)

==== License format ====
* Floating Network License (FNL), "Three-server redundancy"

==== License servers ====
* On client machines, while running the COMSOL installer or license configuration, specify license servers and ports as follows:
clicense1 1718
clicense2 1718
clicense3 1718
: '''Note:''' Use ''short host name'' form exactly as given here. (Technical Reason: The license must be usable both on Carbon's internal network as well as on divisional networks that are outside of Carbon).

'''Advanced:''' A client machine's COMSOL license file <code><COMSOLDIR>/Multiphysics/license/license.dat</code> should look as follows:
<source lang="make">
SERVER clicense1 ANY 1718
SERVER clicense2 ANY 1718
SERVER clicense3 ANY 1718
USE_SERVER
</source>

<code><COMSOLDIR></code> is located by default at:
* '''Windows''': C:\Program files\COMSOL\COMSOL''nn''
* '''Linux''': /usr/local/comsol''nn''
* '''Mac''': /Applications/COMSOL''nn''
("''nn''" is the application's abbreviated version number.)

==== Network Troubleshooting ====
When using the license access model descibed here (as opposed to tunneled over SSH), ensure the following:
* Verify that short host names can be resolved to IP addresses. Either one of the following commands should produce sensible output:
host clicense1
: or:
ping -c 3 clicense1
* Verify that ''both license ports on the same server'' can be reached from your client machine:
telnet '''clicense1''' 1718
telnet '''clicense1''' 1719
: or:
telnet '''clicense2''' 1718
telnet '''clicense2''' 1719
: A ''successful'' connection looks like:
Trying 146.139.x.y...
Connected to clicense1.cnm.anl.gov.
Escape character is '^]'.
:: Do not enter any data. Instead, close each connection by hitting <code>Ctrl</code><code>]</code> (closing square bracket), followed by <code>Ctrl</code><code>D</code>.
: A ''failed'' connection gives one of these errors:
telnet: connect to address 146.139.x.y: Operation timed out
: or:
telnet: connect to address 146.139.x.y: Connection refused
:: In these cases, ask that your IP firewall configuration be reviewed and changed.

=== License configuration when outside CNM and using SSH Tunneling ===
If you're comfortable with ssh and tunneling, configure client licensing as follows:

==== License format ====
* Floating Network License (FNL), "Port number and Hostname"
: '''Note:''' Three-server redundancy cannot be accessed over SSH tunneling. Instead, the user must configure just one server, and if needed in rare situations manually reconfigure to use the other server if the first one fails.

==== License server ====
* Enter into the COMSOL license file:
<source lang="make">
SERVER 127.0.0.1 ANY 1718
USE_SERVER
</source>
: '''Note:''' Do not use "localhost", as this will be directly (and incorrectly) interpreted by COMSOL to mean that the current machine actually ''is'' a license server.

Connect to <code>mega</code> or a Carbon login node with one of the following ssh commands:
* First, set up a tunnel to the primary license server:
ssh -L 1718:'''clicense1''':1718 -L 1719:'''clicense1''':1719 mega
* Start COMSOL.
: If you get an error that the number of licenses is exhausted, that is at least confirmation that you correctly reached the license server.

If the above attempt takes a long time or hangs entirely, it may be because the license service currently runs on the secondary server.
* Connect to the alternate license server:
ssh -L 1718:'''clicense2''':1718 -L 1719:'''clicense2''':1719 mega
* Start COMSOL.

== License Troubleshooting ==
The following are typical error messages.
=== Could not obtain license for COMSOL Multiphysics GUI ===
* Error details:
License error: -4.
'''Licensed number of users already reached.'''
Feature: COMSOLGUI
License path: /Users/stern/Applications/COMSOL52a/license/license.dat:
FlexNet Licensing error:-4,132
* Reason: The connection to the license server succeeded, but the license token(s) you requested are in use.
* Solution: Wait a bit, or [http://carbonmgmt/ganglia/lic.php identify who's using COMSOL] and politely ask the user when they might be finished with their work.

=== License server machine is down or not responding ===
* Error details:
License error: -96.
See the system administrator about starting the license server system, or
make sure you're referring to the right host (see LM_LICENSE_FILE).
Feature: SERIAL
Hostname: clicense3
License path: /Users/stern/Applications/COMSOL52a/license/license.dat:
FlexNet Licensing error:-96,7
* Reason: The license servers cannot be reached at all. The <code>ping</code> or <code>host</code> tests mentioned above fail.
* Solution: Review the client license file and DNS lookup domains.

=== License file does not support this version ===
* Error details:
Feature: SERIAL
'''Application version > License version: 5.21 > 3.5'''
License path: /Users/stern/Applications/COMSOL52a/license/license.dat:
FlexNet Licensing error:-21,126
* Reason: Only the first of two steps for connecting to the license servers succeeds, which may happen when the redundant license service runs on the alternate server instead of the one you're connected to.
* Solution: When using SSH, try <code>clicense2</code> instead of <code>clicense1</code>, or vice versa. Revisit your <code>~/.ssh/config</code> file or use the alternate command line as shown in the SSH section above.

== Multiple CPU cores ==
Ways to start multi-core (from http://www.comsol.com/support/knowledgebase/1096/)
comsol -np 8

export COMSOL_NUM_THREADS=8 # automatically provided on ''Carbon''
comsol
* COMSOL does not benefit from hyperthreading (oversubscribing the physical core count).

== Batch jobs ==

* http://www.comsol.com/support/knowledgebase/1001/ .
* [[Media:HPC 2012-05-16 Comsol parallel usage.pdf]] (Modified from http://www.comsol.com/support/knowledgebase/1001/files/4027/cluster_install_linux_42a.pptx – Appendix slide, PBS)
* '''Note:''' MPD is not supported on Carbon (I patched the main comsol script), and no longer necessary at all in Comsol-4.3.

=== Sample Job file ===
See:
$COMSOL_HOME/sample.job

== Important COMSOL Support Articles ==
* Toplevel suport page: https://www.comsol.com/support

=== KB Articles ===
* [https://www.comsol.com/support/knowledgebase/1001 Running COMSOL® in parallel on clusters]
* [https://www.comsol.com/support/knowledgebase/1250 Running parametric sweeps, batch sweeps, and cluster sweeps from the command line]
* [https://www.comsol.com/support/knowledgebase/1255 Reducing the amount of solution data stored in a model]
* [https://www.comsol.com/support/knowledgebase/830 Handling large models]

=== Blog-style articles ===
* [https://www.comsol.com/blogs/the-power-of-the-batch-sweep/ The Power of the Batch Sweep]
* [https://www.comsol.com/blogs/how-to-run-simulations-in-batch-mode-from-the-command-line/ How to Run Simulations in Batch Mode from the Command Line]

HPC/Applications/comsol

2025-08-07T15:26:59Z

Stern: /* COMSOL Multiphysics */

== Introduction ==
The [http://www.comsol.com/products/multiphysics/ '''COMSOL Multiphysics'''] application suite
is available for Mac, Windows, and Linux.

== Access ==
[[Image:HPC Comsol licensed modules.png|right|450px|thumb|COMSOL features licensed at CNM as of 2024-10.]]
=== COMSOL Multiphysics ===
Access to '''COMSOL Multiphysics''', the main application with both a graphical user interface and batch computing capabilities, will be granted '''''upon request''''' .

You must be:
* An '''''Argonne employee'''''  in one of the following roles:
** '''''CNM Facility User'''''  with an active proposal that has COMSOL in scope — please see [[HPC/FAQ#Access_and_proposals]].
** '''''NST staff'''''  (including PostDocs, joint appointments, and students).

The set of available COMSOL modules that we have licensed is shown at right.

* To have a native desktop version installed on your Argonne-owned computer (laptop or desktop), please contact [mailto:help@anl.gov?Subject=COMSOL%20license%20access%20request CNM support]. Make sure to review [http://www.comsol.com/support/updates/ Product Updates].
* To run COMSOL on our HPC system ''Carbon'', as GUI or as batch computing jobs, your proposal must have ''Carbon'' access in scope.

=== COMSOL Server ===
''CNM Facility Users'' who are not Argonne employees are not eligible to access
the full '''COMSOL Multiphysics''' product, in compliance with its licensing terms and conditions.
However, the [https://www.comsol.com/comsol-server '''COMSOL Server'''] product offers an alternative
albeit more narrow means of access.

All users, including '''''non-Argonne users''''' may be able to run, over a web interface in their browser,
pre-parameterized "fixed-model" simulations that were made with the '''COMSOL Application Builder''' (requires Windows).

To inquire about using COMSOL server applications,
please contact us.

We will review possible purchase paths for COMSOL Server, and the ''considerable effort'' likely required by users and the CNM to develop and deploy a COMSOL application.

== Support ==
* For questions on how to run or start COMSOL:
*: Submit an [[HPC/Support | '''IT support request''']]
* For questions related to most COMSOL data or scripts, contact COMSOL support at:
*: [mailto:support@comsol.com '''support'''@comsol.com]
* '''Exception:''' If your support question might contain data that is subject to '''US Export Control''' regulations, contact instead:
*: [mailto:us-export-control@us.comsol.com '''us-export-control'''@'''us'''.comsol.com]

== The modes to run COMSOL ==
There are several ways of running COMSOL.

=== Running the desktop application ===
To edit models and analyze results, use the COMSOL graphical user interface (GUI). You can run it in several ways, each explained in more detail in the following sections.

# Run on Carbon, display on your desktop using the [http://en.wikipedia.org/wiki/X_Window_System X Window system (X11)].
#* Requires an ''X11 Server'' application running on your client machine, and SSH X11 forwarding.
#* The X11 protocol can be slow over LAN.
# Run on Carbon within a [[HPC/Network Access/Virtual Desktop | virtual desktop environment (VNC)]].
#* Requires a VNC client on your machine.
#* It can be confusing where VNC and the app runs.
#* The virtual desktop presents a (deliberatly) sparse windowing environment – Copy&Paste can be difficult to use.
# Run on your own computer, remotely checking out Carbon's license.
#* Requires a full installation of COMSOL on your desktoo or laptop computer.
#* You will require access to the COMSOL license server running on Carbon, either through VPN (fairly easy) or SSH tunneling (can be intimidating for first-time users). See below for details.

=== Running a batch calculation ===
When the computational stage of models takes longer than a few minutes to complete, it will be beneficial to submit save the model file and run it in parallel on Carbon.
See the [[#Batch jobs|section below]].

== Use with X11 ==
For casual use, and with [[HPC/Network Access/SSH Tunnel Setup on Linux and MacOS | X11 tunneling to Carbon]] set up, simply add to your <code>~/.bashrc</code> file on Carbon:
module load comsol

Then, from the command line on a Carbon login node, start the COMSOL GUI:
comsol &

License configuration is automatic on Carbon.

Comsol will use a number of cores to run calculations multithreaded by default. (On Carbon, COMSOL_NUM_THREADS is set.)

'''Quit the application''' as soon as you are done. Only one user can use Comsol at a time.

== Use with VNC ==
See general [[HPC/Network Access/Virtual Desktop | instructions on using VNC]], and the COMSOL example therein.

== Native desktop version ==

To get the fastest response from the COMSOL GUI, run a native COMSOL version on your desktop machine or laptop, called ''client machine'' below.

Enter license details either in the COMSOL Setup configuration dialog.

== Native desktop version — License configuration ==
A client's COMSOL license needs to be configured depending on how the client reaches CNM network resources. Follow the appropriate section:
* [[#License configuration when inside CNM or using VPN]]
* [[#License configuration when outside CNM and using SSH Tunneling]]

=== License configuration when inside CNM or using VPN ===
==== Ensure correct DNS search domains ====

* Inspect the client machine's network configuration.
* Ensure that '''two''' entries are listed as DNS Search Domains for the current network profile:
nst.anl.gov
cnm.anl.gov
: (The order of entries does not matter for the purposes considered here.)
* If the client machine is ''wired to CNM networks'', the domains are normally provided correctly by DHCP.
* If the client machine is ''connected over VPN'', you may need edit the respective network profile.
:* On a Mac, find the VPN configuration details under System Preferences, Network, Advanced… . You may need to unlock the Network panel first before you can make changes under "Advanced…". It appears that those changes are being saved and will be activated automatically the next time that this VPN session gets started.
: [[Image:VNC_add_cnm_to_DNS_1.png|400px]] [[Image:VNC_add_cnm_to_DNS_2.png|400px]]
:* On Windows or Linux machines, take ''analogous'' steps, using the platform's own configuration tools.

* Exclude license manager components from client desktop installation. (Those are all handled server-side.)

==== License format ====
* Floating Network License (FNL), "Three-server redundancy"

==== License servers ====
* On client machines, while running the COMSOL installer or license configuration, specify license servers and ports as follows:
clicense1 1718
clicense2 1718
clicense3 1718
: '''Note:''' Use ''short host name'' form exactly as given here. (Technical Reason: The license must be usable both on Carbon's internal network as well as on divisional networks that are outside of Carbon).

'''Advanced:''' A client machine's COMSOL license file <code><COMSOLDIR>/Multiphysics/license/license.dat</code> should look as follows:
<source lang="make">
SERVER clicense1 ANY 1718
SERVER clicense2 ANY 1718
SERVER clicense3 ANY 1718
USE_SERVER
</source>

<code><COMSOLDIR></code> is located by default at:
* '''Windows''': C:\Program files\COMSOL\COMSOL''nn''
* '''Linux''': /usr/local/comsol''nn''
* '''Mac''': /Applications/COMSOL''nn''
("''nn''" is the application's abbreviated version number.)

==== Network Troubleshooting ====
When using the license access model descibed here (as opposed to tunneled over SSH), ensure the following:
* Verify that short host names can be resolved to IP addresses. Either one of the following commands should produce sensible output:
host clicense1
: or:
ping -c 3 clicense1
* Verify that ''both license ports on the same server'' can be reached from your client machine:
telnet '''clicense1''' 1718
telnet '''clicense1''' 1719
: or:
telnet '''clicense2''' 1718
telnet '''clicense2''' 1719
: A ''successful'' connection looks like:
Trying 146.139.x.y...
Connected to clicense1.cnm.anl.gov.
Escape character is '^]'.
:: Do not enter any data. Instead, close each connection by hitting <code>Ctrl</code><code>]</code> (closing square bracket), followed by <code>Ctrl</code><code>D</code>.
: A ''failed'' connection gives one of these errors:
telnet: connect to address 146.139.x.y: Operation timed out
: or:
telnet: connect to address 146.139.x.y: Connection refused
:: In these cases, ask that your IP firewall configuration be reviewed and changed.

=== License configuration when outside CNM and using SSH Tunneling ===
If you're comfortable with ssh and tunneling, configure client licensing as follows:

==== License format ====
* Floating Network License (FNL), "Port number and Hostname"
: '''Note:''' Three-server redundancy cannot be accessed over SSH tunneling. Instead, the user must configure just one server, and if needed in rare situations manually reconfigure to use the other server if the first one fails.

==== License server ====
* Enter into the COMSOL license file:
<source lang="make">
SERVER 127.0.0.1 ANY 1718
USE_SERVER
</source>
: '''Note:''' Do not use "localhost", as this will be directly (and incorrectly) interpreted by COMSOL to mean that the current machine actually ''is'' a license server.

Connect to <code>mega</code> or a Carbon login node with one of the following ssh commands:
* First, set up a tunnel to the primary license server:
ssh -L 1718:'''clicense1''':1718 -L 1719:'''clicense1''':1719 mega
* Start COMSOL.
: If you get an error that the number of licenses is exhausted, that is at least confirmation that you correctly reached the license server.

If the above attempt takes a long time or hangs entirely, it may be because the license service currently runs on the secondary server.
* Connect to the alternate license server:
ssh -L 1718:'''clicense2''':1718 -L 1719:'''clicense2''':1719 mega
* Start COMSOL.

== License Troubleshooting ==
The following are typical error messages.
=== Could not obtain license for COMSOL Multiphysics GUI ===
* Error details:
License error: -4.
'''Licensed number of users already reached.'''
Feature: COMSOLGUI
License path: /Users/stern/Applications/COMSOL52a/license/license.dat:
FlexNet Licensing error:-4,132
* Reason: The connection to the license server succeeded, but the license token(s) you requested are in use.
* Solution: Wait a bit, or [http://carbonmgmt/ganglia/lic.php identify who's using COMSOL] and politely ask the user when they might be finished with their work.

=== License server machine is down or not responding ===
* Error details:
License error: -96.
See the system administrator about starting the license server system, or
make sure you're referring to the right host (see LM_LICENSE_FILE).
Feature: SERIAL
Hostname: clicense3
License path: /Users/stern/Applications/COMSOL52a/license/license.dat:
FlexNet Licensing error:-96,7
* Reason: The license servers cannot be reached at all. The <code>ping</code> or <code>host</code> tests mentioned above fail.
* Solution: Review the client license file and DNS lookup domains.

=== License file does not support this version ===
* Error details:
Feature: SERIAL
'''Application version > License version: 5.21 > 3.5'''
License path: /Users/stern/Applications/COMSOL52a/license/license.dat:
FlexNet Licensing error:-21,126
* Reason: Only the first of two steps for connecting to the license servers succeeds, which may happen when the redundant license service runs on the alternate server instead of the one you're connected to.
* Solution: When using SSH, try <code>clicense2</code> instead of <code>clicense1</code>, or vice versa. Revisit your <code>~/.ssh/config</code> file or use the alternate command line as shown in the SSH section above.

== Multiple CPU cores ==
Ways to start multi-core (from http://www.comsol.com/support/knowledgebase/1096/)
comsol -np 8

export COMSOL_NUM_THREADS=8 # automatically provided on ''Carbon''
comsol
* COMSOL does not benefit from hyperthreading (oversubscribing the physical core count).

== Batch jobs ==

* http://www.comsol.com/support/knowledgebase/1001/ .
* [[Media:HPC 2012-05-16 Comsol parallel usage.pdf]] (Modified from http://www.comsol.com/support/knowledgebase/1001/files/4027/cluster_install_linux_42a.pptx – Appendix slide, PBS)
* '''Note:''' MPD is not supported on Carbon (I patched the main comsol script), and no longer necessary at all in Comsol-4.3.

=== Sample Job file ===
See:
$COMSOL_HOME/sample.job

== Important COMSOL Support Articles ==
* Toplevel suport page: https://www.comsol.com/support

=== KB Articles ===
* [https://www.comsol.com/support/knowledgebase/1001 Running COMSOL® in parallel on clusters]
* [https://www.comsol.com/support/knowledgebase/1250 Running parametric sweeps, batch sweeps, and cluster sweeps from the command line]
* [https://www.comsol.com/support/knowledgebase/1255 Reducing the amount of solution data stored in a model]
* [https://www.comsol.com/support/knowledgebase/830 Handling large models]

=== Blog-style articles ===
* [https://www.comsol.com/blogs/the-power-of-the-batch-sweep/ The Power of the Batch Sweep]
* [https://www.comsol.com/blogs/how-to-run-simulations-in-batch-mode-from-the-command-line/ How to Run Simulations in Batch Mode from the Command Line]

HPC/Applications/comsol

2025-08-07T15:26:41Z

Stern:

== Introduction ==
The [http://www.comsol.com/products/multiphysics/ '''COMSOL Multiphysics'''] application suite
is available for Mac, Windows, and Linux.

== Access ==
[[Image:HPC Comsol licensed modules.png|right|450px|thumb|COMSOL features licensed at CNM as of 2024-10.]]
=== COMSOL Multiphysics ===
Access to '''COMSOL Multiphysics''', the main application with both a graphical user interface and batch computing capabilities, will be granted '''''upon request''''' .

You must be:
* An '''''Argonne employee'''''  in of the following roles:
** '''''CNM Facility User'''''  with an active proposal that has COMSOL in scope — please see [[HPC/FAQ#Access_and_proposals]].
** '''''NST staff'''''  (including PostDocs, joint appointments, and students).

The set of available COMSOL modules that we have licensed is shown at right.

* To have a native desktop version installed on your Argonne-owned computer (laptop or desktop), please contact [mailto:help@anl.gov?Subject=COMSOL%20license%20access%20request CNM support]. Make sure to review [http://www.comsol.com/support/updates/ Product Updates].
* To run COMSOL on our HPC system ''Carbon'', as GUI or as batch computing jobs, your proposal must have ''Carbon'' access in scope.

=== COMSOL Server ===
''CNM Facility Users'' who are not Argonne employees are not eligible to access
the full '''COMSOL Multiphysics''' product, in compliance with its licensing terms and conditions.
However, the [https://www.comsol.com/comsol-server '''COMSOL Server'''] product offers an alternative
albeit more narrow means of access.

All users, including '''''non-Argonne users''''' may be able to run, over a web interface in their browser,
pre-parameterized "fixed-model" simulations that were made with the '''COMSOL Application Builder''' (requires Windows).

To inquire about using COMSOL server applications,
please contact us.

We will review possible purchase paths for COMSOL Server, and the ''considerable effort'' likely required by users and the CNM to develop and deploy a COMSOL application.

== Support ==
* For questions on how to run or start COMSOL:
*: Submit an [[HPC/Support | '''IT support request''']]
* For questions related to most COMSOL data or scripts, contact COMSOL support at:
*: [mailto:support@comsol.com '''support'''@comsol.com]
* '''Exception:''' If your support question might contain data that is subject to '''US Export Control''' regulations, contact instead:
*: [mailto:us-export-control@us.comsol.com '''us-export-control'''@'''us'''.comsol.com]

== The modes to run COMSOL ==
There are several ways of running COMSOL.

=== Running the desktop application ===
To edit models and analyze results, use the COMSOL graphical user interface (GUI). You can run it in several ways, each explained in more detail in the following sections.

# Run on Carbon, display on your desktop using the [http://en.wikipedia.org/wiki/X_Window_System X Window system (X11)].
#* Requires an ''X11 Server'' application running on your client machine, and SSH X11 forwarding.
#* The X11 protocol can be slow over LAN.
# Run on Carbon within a [[HPC/Network Access/Virtual Desktop | virtual desktop environment (VNC)]].
#* Requires a VNC client on your machine.
#* It can be confusing where VNC and the app runs.
#* The virtual desktop presents a (deliberatly) sparse windowing environment – Copy&Paste can be difficult to use.
# Run on your own computer, remotely checking out Carbon's license.
#* Requires a full installation of COMSOL on your desktoo or laptop computer.
#* You will require access to the COMSOL license server running on Carbon, either through VPN (fairly easy) or SSH tunneling (can be intimidating for first-time users). See below for details.

=== Running a batch calculation ===
When the computational stage of models takes longer than a few minutes to complete, it will be beneficial to submit save the model file and run it in parallel on Carbon.
See the [[#Batch jobs|section below]].

== Use with X11 ==
For casual use, and with [[HPC/Network Access/SSH Tunnel Setup on Linux and MacOS | X11 tunneling to Carbon]] set up, simply add to your <code>~/.bashrc</code> file on Carbon:
module load comsol

Then, from the command line on a Carbon login node, start the COMSOL GUI:
comsol &

License configuration is automatic on Carbon.

Comsol will use a number of cores to run calculations multithreaded by default. (On Carbon, COMSOL_NUM_THREADS is set.)

'''Quit the application''' as soon as you are done. Only one user can use Comsol at a time.

== Use with VNC ==
See general [[HPC/Network Access/Virtual Desktop | instructions on using VNC]], and the COMSOL example therein.

== Native desktop version ==

To get the fastest response from the COMSOL GUI, run a native COMSOL version on your desktop machine or laptop, called ''client machine'' below.

Enter license details either in the COMSOL Setup configuration dialog.

== Native desktop version — License configuration ==
A client's COMSOL license needs to be configured depending on how the client reaches CNM network resources. Follow the appropriate section:
* [[#License configuration when inside CNM or using VPN]]
* [[#License configuration when outside CNM and using SSH Tunneling]]

=== License configuration when inside CNM or using VPN ===
==== Ensure correct DNS search domains ====

* Inspect the client machine's network configuration.
* Ensure that '''two''' entries are listed as DNS Search Domains for the current network profile:
nst.anl.gov
cnm.anl.gov
: (The order of entries does not matter for the purposes considered here.)
* If the client machine is ''wired to CNM networks'', the domains are normally provided correctly by DHCP.
* If the client machine is ''connected over VPN'', you may need edit the respective network profile.
:* On a Mac, find the VPN configuration details under System Preferences, Network, Advanced… . You may need to unlock the Network panel first before you can make changes under "Advanced…". It appears that those changes are being saved and will be activated automatically the next time that this VPN session gets started.
: [[Image:VNC_add_cnm_to_DNS_1.png|400px]] [[Image:VNC_add_cnm_to_DNS_2.png|400px]]
:* On Windows or Linux machines, take ''analogous'' steps, using the platform's own configuration tools.

* Exclude license manager components from client desktop installation. (Those are all handled server-side.)

==== License format ====
* Floating Network License (FNL), "Three-server redundancy"

==== License servers ====
* On client machines, while running the COMSOL installer or license configuration, specify license servers and ports as follows:
clicense1 1718
clicense2 1718
clicense3 1718
: '''Note:''' Use ''short host name'' form exactly as given here. (Technical Reason: The license must be usable both on Carbon's internal network as well as on divisional networks that are outside of Carbon).

'''Advanced:''' A client machine's COMSOL license file <code><COMSOLDIR>/Multiphysics/license/license.dat</code> should look as follows:
<source lang="make">
SERVER clicense1 ANY 1718
SERVER clicense2 ANY 1718
SERVER clicense3 ANY 1718
USE_SERVER
</source>

<code><COMSOLDIR></code> is located by default at:
* '''Windows''': C:\Program files\COMSOL\COMSOL''nn''
* '''Linux''': /usr/local/comsol''nn''
* '''Mac''': /Applications/COMSOL''nn''
("''nn''" is the application's abbreviated version number.)

==== Network Troubleshooting ====
When using the license access model descibed here (as opposed to tunneled over SSH), ensure the following:
* Verify that short host names can be resolved to IP addresses. Either one of the following commands should produce sensible output:
host clicense1
: or:
ping -c 3 clicense1
* Verify that ''both license ports on the same server'' can be reached from your client machine:
telnet '''clicense1''' 1718
telnet '''clicense1''' 1719
: or:
telnet '''clicense2''' 1718
telnet '''clicense2''' 1719
: A ''successful'' connection looks like:
Trying 146.139.x.y...
Connected to clicense1.cnm.anl.gov.
Escape character is '^]'.
:: Do not enter any data. Instead, close each connection by hitting <code>Ctrl</code><code>]</code> (closing square bracket), followed by <code>Ctrl</code><code>D</code>.
: A ''failed'' connection gives one of these errors:
telnet: connect to address 146.139.x.y: Operation timed out
: or:
telnet: connect to address 146.139.x.y: Connection refused
:: In these cases, ask that your IP firewall configuration be reviewed and changed.

=== License configuration when outside CNM and using SSH Tunneling ===
If you're comfortable with ssh and tunneling, configure client licensing as follows:

==== License format ====
* Floating Network License (FNL), "Port number and Hostname"
: '''Note:''' Three-server redundancy cannot be accessed over SSH tunneling. Instead, the user must configure just one server, and if needed in rare situations manually reconfigure to use the other server if the first one fails.

==== License server ====
* Enter into the COMSOL license file:
<source lang="make">
SERVER 127.0.0.1 ANY 1718
USE_SERVER
</source>
: '''Note:''' Do not use "localhost", as this will be directly (and incorrectly) interpreted by COMSOL to mean that the current machine actually ''is'' a license server.

Connect to <code>mega</code> or a Carbon login node with one of the following ssh commands:
* First, set up a tunnel to the primary license server:
ssh -L 1718:'''clicense1''':1718 -L 1719:'''clicense1''':1719 mega
* Start COMSOL.
: If you get an error that the number of licenses is exhausted, that is at least confirmation that you correctly reached the license server.

If the above attempt takes a long time or hangs entirely, it may be because the license service currently runs on the secondary server.
* Connect to the alternate license server:
ssh -L 1718:'''clicense2''':1718 -L 1719:'''clicense2''':1719 mega
* Start COMSOL.

== License Troubleshooting ==
The following are typical error messages.
=== Could not obtain license for COMSOL Multiphysics GUI ===
* Error details:
License error: -4.
'''Licensed number of users already reached.'''
Feature: COMSOLGUI
License path: /Users/stern/Applications/COMSOL52a/license/license.dat:
FlexNet Licensing error:-4,132
* Reason: The connection to the license server succeeded, but the license token(s) you requested are in use.
* Solution: Wait a bit, or [http://carbonmgmt/ganglia/lic.php identify who's using COMSOL] and politely ask the user when they might be finished with their work.

=== License server machine is down or not responding ===
* Error details:
License error: -96.
See the system administrator about starting the license server system, or
make sure you're referring to the right host (see LM_LICENSE_FILE).
Feature: SERIAL
Hostname: clicense3
License path: /Users/stern/Applications/COMSOL52a/license/license.dat:
FlexNet Licensing error:-96,7
* Reason: The license servers cannot be reached at all. The <code>ping</code> or <code>host</code> tests mentioned above fail.
* Solution: Review the client license file and DNS lookup domains.

=== License file does not support this version ===
* Error details:
Feature: SERIAL
'''Application version > License version: 5.21 > 3.5'''
License path: /Users/stern/Applications/COMSOL52a/license/license.dat:
FlexNet Licensing error:-21,126
* Reason: Only the first of two steps for connecting to the license servers succeeds, which may happen when the redundant license service runs on the alternate server instead of the one you're connected to.
* Solution: When using SSH, try <code>clicense2</code> instead of <code>clicense1</code>, or vice versa. Revisit your <code>~/.ssh/config</code> file or use the alternate command line as shown in the SSH section above.

== Multiple CPU cores ==
Ways to start multi-core (from http://www.comsol.com/support/knowledgebase/1096/)
comsol -np 8

export COMSOL_NUM_THREADS=8 # automatically provided on ''Carbon''
comsol
* COMSOL does not benefit from hyperthreading (oversubscribing the physical core count).

== Batch jobs ==

* http://www.comsol.com/support/knowledgebase/1001/ .
* [[Media:HPC 2012-05-16 Comsol parallel usage.pdf]] (Modified from http://www.comsol.com/support/knowledgebase/1001/files/4027/cluster_install_linux_42a.pptx – Appendix slide, PBS)
* '''Note:''' MPD is not supported on Carbon (I patched the main comsol script), and no longer necessary at all in Comsol-4.3.

=== Sample Job file ===
See:
$COMSOL_HOME/sample.job

== Important COMSOL Support Articles ==
* Toplevel suport page: https://www.comsol.com/support

=== KB Articles ===
* [https://www.comsol.com/support/knowledgebase/1001 Running COMSOL® in parallel on clusters]
* [https://www.comsol.com/support/knowledgebase/1250 Running parametric sweeps, batch sweeps, and cluster sweeps from the command line]
* [https://www.comsol.com/support/knowledgebase/1255 Reducing the amount of solution data stored in a model]
* [https://www.comsol.com/support/knowledgebase/830 Handling large models]

=== Blog-style articles ===
* [https://www.comsol.com/blogs/the-power-of-the-batch-sweep/ The Power of the Batch Sweep]
* [https://www.comsol.com/blogs/how-to-run-simulations-in-batch-mode-from-the-command-line/ How to Run Simulations in Batch Mode from the Command Line]

HPC/Applications/comsol

2025-08-07T15:25:07Z

Stern: /* COMSOL Multiphysics */

== Introduction ==
The [http://www.comsol.com/products/multiphysics/ '''COMSOL Multiphysics'''] application suite
is available for Mac, Windows, and Linux.
[[Image:HPC Comsol licensed modules.png|right|450px|thumb|COMSOL features licensed at CNM as of 2024-10.]]

== Access ==
=== COMSOL Multiphysics ===
Access to '''COMSOL Multiphysics''', the main application with both a graphical user interface and batch computing capabilities, will be granted '''''upon request''''' .

You must be:
* An '''''Argonne employee'''''  in of the following roles:
** '''''CNM Facility User'''''  with an active proposal with COMSOL in scope — please see [[HPC/FAQ#Access_and_proposals]].
** '''''NST staff'''''  (including PostDocs, joint appointments, and students).

The set of available COMSOL modules that we have licensed is shown at right.

* To have a native desktop version installed on your Argonne-owned computer (laptop or desktop), please contact [mailto:help@anl.gov?Subject=COMSOL%20license%20access%20request CNM support]. Make sure to review [http://www.comsol.com/support/updates/ Product Updates].
* To run COMSOL on our HPC system ''Carbon'', as GUI or as batch computing jobs, your proposal must have ''Carbon'' access in scope.

=== COMSOL Server ===
''CNM Facility Users'' who are not Argonne employees are not eligible to access
the full '''COMSOL Multiphysics''' product, in compliance with its licensing terms and conditions.
However, the [https://www.comsol.com/comsol-server '''COMSOL Server'''] product offers an alternative
albeit more narrow means of access.

All users, including '''''non-Argonne users''''' may be able to run, over a web interface in their browser,
pre-parameterized "fixed-model" simulations that were made with the '''COMSOL Application Builder''' (requires Windows).

To inquire about using COMSOL server applications,
please contact us.

We will review possible purchase paths for COMSOL Server, and the ''considerable effort'' likely required by users and the CNM to develop and deploy a COMSOL application.

== Support ==
* For questions on how to run or start COMSOL:
*: Submit an [[HPC/Support | '''IT support request''']]
* For questions related to most COMSOL data or scripts, contact COMSOL support at:
*: [mailto:support@comsol.com '''support'''@comsol.com]
* '''Exception:''' If your support question might contain data that is subject to '''US Export Control''' regulations, contact instead:
*: [mailto:us-export-control@us.comsol.com '''us-export-control'''@'''us'''.comsol.com]

== The modes to run COMSOL ==
There are several ways of running COMSOL.

=== Running the desktop application ===
To edit models and analyze results, use the COMSOL graphical user interface (GUI). You can run it in several ways, each explained in more detail in the following sections.

# Run on Carbon, display on your desktop using the [http://en.wikipedia.org/wiki/X_Window_System X Window system (X11)].
#* Requires an ''X11 Server'' application running on your client machine, and SSH X11 forwarding.
#* The X11 protocol can be slow over LAN.
# Run on Carbon within a [[HPC/Network Access/Virtual Desktop | virtual desktop environment (VNC)]].
#* Requires a VNC client on your machine.
#* It can be confusing where VNC and the app runs.
#* The virtual desktop presents a (deliberatly) sparse windowing environment – Copy&Paste can be difficult to use.
# Run on your own computer, remotely checking out Carbon's license.
#* Requires a full installation of COMSOL on your desktoo or laptop computer.
#* You will require access to the COMSOL license server running on Carbon, either through VPN (fairly easy) or SSH tunneling (can be intimidating for first-time users). See below for details.

=== Running a batch calculation ===
When the computational stage of models takes longer than a few minutes to complete, it will be beneficial to submit save the model file and run it in parallel on Carbon.
See the [[#Batch jobs|section below]].

== Use with X11 ==
For casual use, and with [[HPC/Network Access/SSH Tunnel Setup on Linux and MacOS | X11 tunneling to Carbon]] set up, simply add to your <code>~/.bashrc</code> file on Carbon:
module load comsol

Then, from the command line on a Carbon login node, start the COMSOL GUI:
comsol &

License configuration is automatic on Carbon.

Comsol will use a number of cores to run calculations multithreaded by default. (On Carbon, COMSOL_NUM_THREADS is set.)

'''Quit the application''' as soon as you are done. Only one user can use Comsol at a time.

== Use with VNC ==
See general [[HPC/Network Access/Virtual Desktop | instructions on using VNC]], and the COMSOL example therein.

== Native desktop version ==

To get the fastest response from the COMSOL GUI, run a native COMSOL version on your desktop machine or laptop, called ''client machine'' below.

Enter license details either in the COMSOL Setup configuration dialog.

== Native desktop version — License configuration ==
A client's COMSOL license needs to be configured depending on how the client reaches CNM network resources. Follow the appropriate section:
* [[#License configuration when inside CNM or using VPN]]
* [[#License configuration when outside CNM and using SSH Tunneling]]

=== License configuration when inside CNM or using VPN ===
==== Ensure correct DNS search domains ====

* Inspect the client machine's network configuration.
* Ensure that '''two''' entries are listed as DNS Search Domains for the current network profile:
nst.anl.gov
cnm.anl.gov
: (The order of entries does not matter for the purposes considered here.)
* If the client machine is ''wired to CNM networks'', the domains are normally provided correctly by DHCP.
* If the client machine is ''connected over VPN'', you may need edit the respective network profile.
:* On a Mac, find the VPN configuration details under System Preferences, Network, Advanced… . You may need to unlock the Network panel first before you can make changes under "Advanced…". It appears that those changes are being saved and will be activated automatically the next time that this VPN session gets started.
: [[Image:VNC_add_cnm_to_DNS_1.png|400px]] [[Image:VNC_add_cnm_to_DNS_2.png|400px]]
:* On Windows or Linux machines, take ''analogous'' steps, using the platform's own configuration tools.

* Exclude license manager components from client desktop installation. (Those are all handled server-side.)

==== License format ====
* Floating Network License (FNL), "Three-server redundancy"

==== License servers ====
* On client machines, while running the COMSOL installer or license configuration, specify license servers and ports as follows:
clicense1 1718
clicense2 1718
clicense3 1718
: '''Note:''' Use ''short host name'' form exactly as given here. (Technical Reason: The license must be usable both on Carbon's internal network as well as on divisional networks that are outside of Carbon).

'''Advanced:''' A client machine's COMSOL license file <code><COMSOLDIR>/Multiphysics/license/license.dat</code> should look as follows:
<source lang="make">
SERVER clicense1 ANY 1718
SERVER clicense2 ANY 1718
SERVER clicense3 ANY 1718
USE_SERVER
</source>

<code><COMSOLDIR></code> is located by default at:
* '''Windows''': C:\Program files\COMSOL\COMSOL''nn''
* '''Linux''': /usr/local/comsol''nn''
* '''Mac''': /Applications/COMSOL''nn''
("''nn''" is the application's abbreviated version number.)

==== Network Troubleshooting ====
When using the license access model descibed here (as opposed to tunneled over SSH), ensure the following:
* Verify that short host names can be resolved to IP addresses. Either one of the following commands should produce sensible output:
host clicense1
: or:
ping -c 3 clicense1
* Verify that ''both license ports on the same server'' can be reached from your client machine:
telnet '''clicense1''' 1718
telnet '''clicense1''' 1719
: or:
telnet '''clicense2''' 1718
telnet '''clicense2''' 1719
: A ''successful'' connection looks like:
Trying 146.139.x.y...
Connected to clicense1.cnm.anl.gov.
Escape character is '^]'.
:: Do not enter any data. Instead, close each connection by hitting <code>Ctrl</code><code>]</code> (closing square bracket), followed by <code>Ctrl</code><code>D</code>.
: A ''failed'' connection gives one of these errors:
telnet: connect to address 146.139.x.y: Operation timed out
: or:
telnet: connect to address 146.139.x.y: Connection refused
:: In these cases, ask that your IP firewall configuration be reviewed and changed.

=== License configuration when outside CNM and using SSH Tunneling ===
If you're comfortable with ssh and tunneling, configure client licensing as follows:

==== License format ====
* Floating Network License (FNL), "Port number and Hostname"
: '''Note:''' Three-server redundancy cannot be accessed over SSH tunneling. Instead, the user must configure just one server, and if needed in rare situations manually reconfigure to use the other server if the first one fails.

==== License server ====
* Enter into the COMSOL license file:
<source lang="make">
SERVER 127.0.0.1 ANY 1718
USE_SERVER
</source>
: '''Note:''' Do not use "localhost", as this will be directly (and incorrectly) interpreted by COMSOL to mean that the current machine actually ''is'' a license server.

Connect to <code>mega</code> or a Carbon login node with one of the following ssh commands:
* First, set up a tunnel to the primary license server:
ssh -L 1718:'''clicense1''':1718 -L 1719:'''clicense1''':1719 mega
* Start COMSOL.
: If you get an error that the number of licenses is exhausted, that is at least confirmation that you correctly reached the license server.

If the above attempt takes a long time or hangs entirely, it may be because the license service currently runs on the secondary server.
* Connect to the alternate license server:
ssh -L 1718:'''clicense2''':1718 -L 1719:'''clicense2''':1719 mega
* Start COMSOL.

== License Troubleshooting ==
The following are typical error messages.
=== Could not obtain license for COMSOL Multiphysics GUI ===
* Error details:
License error: -4.
'''Licensed number of users already reached.'''
Feature: COMSOLGUI
License path: /Users/stern/Applications/COMSOL52a/license/license.dat:
FlexNet Licensing error:-4,132
* Reason: The connection to the license server succeeded, but the license token(s) you requested are in use.
* Solution: Wait a bit, or [http://carbonmgmt/ganglia/lic.php identify who's using COMSOL] and politely ask the user when they might be finished with their work.

=== License server machine is down or not responding ===
* Error details:
License error: -96.
See the system administrator about starting the license server system, or
make sure you're referring to the right host (see LM_LICENSE_FILE).
Feature: SERIAL
Hostname: clicense3
License path: /Users/stern/Applications/COMSOL52a/license/license.dat:
FlexNet Licensing error:-96,7
* Reason: The license servers cannot be reached at all. The <code>ping</code> or <code>host</code> tests mentioned above fail.
* Solution: Review the client license file and DNS lookup domains.

=== License file does not support this version ===
* Error details:
Feature: SERIAL
'''Application version > License version: 5.21 > 3.5'''
License path: /Users/stern/Applications/COMSOL52a/license/license.dat:
FlexNet Licensing error:-21,126
* Reason: Only the first of two steps for connecting to the license servers succeeds, which may happen when the redundant license service runs on the alternate server instead of the one you're connected to.
* Solution: When using SSH, try <code>clicense2</code> instead of <code>clicense1</code>, or vice versa. Revisit your <code>~/.ssh/config</code> file or use the alternate command line as shown in the SSH section above.

== Multiple CPU cores ==
Ways to start multi-core (from http://www.comsol.com/support/knowledgebase/1096/)
comsol -np 8

export COMSOL_NUM_THREADS=8 # automatically provided on ''Carbon''
comsol
* COMSOL does not benefit from hyperthreading (oversubscribing the physical core count).

== Batch jobs ==

* http://www.comsol.com/support/knowledgebase/1001/ .
* [[Media:HPC 2012-05-16 Comsol parallel usage.pdf]] (Modified from http://www.comsol.com/support/knowledgebase/1001/files/4027/cluster_install_linux_42a.pptx – Appendix slide, PBS)
* '''Note:''' MPD is not supported on Carbon (I patched the main comsol script), and no longer necessary at all in Comsol-4.3.

=== Sample Job file ===
See:
$COMSOL_HOME/sample.job

== Important COMSOL Support Articles ==
* Toplevel suport page: https://www.comsol.com/support

=== KB Articles ===
* [https://www.comsol.com/support/knowledgebase/1001 Running COMSOL® in parallel on clusters]
* [https://www.comsol.com/support/knowledgebase/1250 Running parametric sweeps, batch sweeps, and cluster sweeps from the command line]
* [https://www.comsol.com/support/knowledgebase/1255 Reducing the amount of solution data stored in a model]
* [https://www.comsol.com/support/knowledgebase/830 Handling large models]

=== Blog-style articles ===
* [https://www.comsol.com/blogs/the-power-of-the-batch-sweep/ The Power of the Batch Sweep]
* [https://www.comsol.com/blogs/how-to-run-simulations-in-batch-mode-from-the-command-line/ How to Run Simulations in Batch Mode from the Command Line]

HPC/Applications/comsol

2025-08-07T15:11:39Z

Stern:

== Introduction ==
The [http://www.comsol.com/products/multiphysics/ '''COMSOL Multiphysics'''] application suite
is available for Mac, Windows, and Linux.
[[Image:HPC Comsol licensed modules.png|right|450px|thumb|COMSOL features licensed at CNM as of 2024-10.]]

== Access ==
=== COMSOL Multiphysics ===
Access to COMSOL's main graphical user interface application and its batch computing capabilities
can be granted
'''''upon request''''' 
to
'''''Argonne employees''''' 
in the following roles:
* ''CNM Facility Users'', under an active proposal with COMSOL in scope — please see [[HPC/FAQ#Access_and_proposals]].
* ''NST staff'' (including PostDocs, joint appointments, and students).

Of the available features, we have licensed a subset, shown at right.
* To have a native desktop version installed on your Argonne-owned computer (laptop or desktop), please contact [mailto:help@anl.gov?Subject=COMSOL%20license%20access%20request CNM support]. Make sure to review [http://www.comsol.com/support/updates/ Product Updates].
* To run COMSOL on Carbon itself, as GUI or as batch computing jobs, your proposal must have Carbon access in scope.

=== COMSOL Server ===
''CNM Facility Users'' who are not Argonne employees are not eligible to access
the full '''COMSOL Multiphysics''' product, in compliance with its licensing terms and conditions.
However, the [https://www.comsol.com/comsol-server '''COMSOL Server'''] product offers an alternative
albeit more narrow means of access.

All users, including '''''non-Argonne users''''' may be able to run, over a web interface in their browser,
pre-parameterized "fixed-model" simulations that were made with the '''COMSOL Application Builder''' (requires Windows).

To inquire about using COMSOL server applications,
please contact us.

We will review possible purchase paths for COMSOL Server, and the ''considerable effort'' likely required by users and the CNM to develop and deploy a COMSOL application.

== Support ==
* For questions on how to run or start COMSOL:
*: Submit an [[HPC/Support | '''IT support request''']]
* For questions related to most COMSOL data or scripts, contact COMSOL support at:
*: [mailto:support@comsol.com '''support'''@comsol.com]
* '''Exception:''' If your support question might contain data that is subject to '''US Export Control''' regulations, contact instead:
*: [mailto:us-export-control@us.comsol.com '''us-export-control'''@'''us'''.comsol.com]

== The modes to run COMSOL ==
There are several ways of running COMSOL.

=== Running the desktop application ===
To edit models and analyze results, use the COMSOL graphical user interface (GUI). You can run it in several ways, each explained in more detail in the following sections.

# Run on Carbon, display on your desktop using the [http://en.wikipedia.org/wiki/X_Window_System X Window system (X11)].
#* Requires an ''X11 Server'' application running on your client machine, and SSH X11 forwarding.
#* The X11 protocol can be slow over LAN.
# Run on Carbon within a [[HPC/Network Access/Virtual Desktop | virtual desktop environment (VNC)]].
#* Requires a VNC client on your machine.
#* It can be confusing where VNC and the app runs.
#* The virtual desktop presents a (deliberatly) sparse windowing environment – Copy&Paste can be difficult to use.
# Run on your own computer, remotely checking out Carbon's license.
#* Requires a full installation of COMSOL on your desktoo or laptop computer.
#* You will require access to the COMSOL license server running on Carbon, either through VPN (fairly easy) or SSH tunneling (can be intimidating for first-time users). See below for details.

=== Running a batch calculation ===
When the computational stage of models takes longer than a few minutes to complete, it will be beneficial to submit save the model file and run it in parallel on Carbon.
See the [[#Batch jobs|section below]].

== Use with X11 ==
For casual use, and with [[HPC/Network Access/SSH Tunnel Setup on Linux and MacOS | X11 tunneling to Carbon]] set up, simply add to your <code>~/.bashrc</code> file on Carbon:
module load comsol

Then, from the command line on a Carbon login node, start the COMSOL GUI:
comsol &

License configuration is automatic on Carbon.

Comsol will use a number of cores to run calculations multithreaded by default. (On Carbon, COMSOL_NUM_THREADS is set.)

'''Quit the application''' as soon as you are done. Only one user can use Comsol at a time.

== Use with VNC ==
See general [[HPC/Network Access/Virtual Desktop | instructions on using VNC]], and the COMSOL example therein.

== Native desktop version ==

To get the fastest response from the COMSOL GUI, run a native COMSOL version on your desktop machine or laptop, called ''client machine'' below.

Enter license details either in the COMSOL Setup configuration dialog.

== Native desktop version — License configuration ==
A client's COMSOL license needs to be configured depending on how the client reaches CNM network resources. Follow the appropriate section:
* [[#License configuration when inside CNM or using VPN]]
* [[#License configuration when outside CNM and using SSH Tunneling]]

=== License configuration when inside CNM or using VPN ===
==== Ensure correct DNS search domains ====

* Inspect the client machine's network configuration.
* Ensure that '''two''' entries are listed as DNS Search Domains for the current network profile:
nst.anl.gov
cnm.anl.gov
: (The order of entries does not matter for the purposes considered here.)
* If the client machine is ''wired to CNM networks'', the domains are normally provided correctly by DHCP.
* If the client machine is ''connected over VPN'', you may need edit the respective network profile.
:* On a Mac, find the VPN configuration details under System Preferences, Network, Advanced… . You may need to unlock the Network panel first before you can make changes under "Advanced…". It appears that those changes are being saved and will be activated automatically the next time that this VPN session gets started.
: [[Image:VNC_add_cnm_to_DNS_1.png|400px]] [[Image:VNC_add_cnm_to_DNS_2.png|400px]]
:* On Windows or Linux machines, take ''analogous'' steps, using the platform's own configuration tools.

* Exclude license manager components from client desktop installation. (Those are all handled server-side.)

==== License format ====
* Floating Network License (FNL), "Three-server redundancy"

==== License servers ====
* On client machines, while running the COMSOL installer or license configuration, specify license servers and ports as follows:
clicense1 1718
clicense2 1718
clicense3 1718
: '''Note:''' Use ''short host name'' form exactly as given here. (Technical Reason: The license must be usable both on Carbon's internal network as well as on divisional networks that are outside of Carbon).

'''Advanced:''' A client machine's COMSOL license file <code><COMSOLDIR>/Multiphysics/license/license.dat</code> should look as follows:
<source lang="make">
SERVER clicense1 ANY 1718
SERVER clicense2 ANY 1718
SERVER clicense3 ANY 1718
USE_SERVER
</source>

<code><COMSOLDIR></code> is located by default at:
* '''Windows''': C:\Program files\COMSOL\COMSOL''nn''
* '''Linux''': /usr/local/comsol''nn''
* '''Mac''': /Applications/COMSOL''nn''
("''nn''" is the application's abbreviated version number.)

==== Network Troubleshooting ====
When using the license access model descibed here (as opposed to tunneled over SSH), ensure the following:
* Verify that short host names can be resolved to IP addresses. Either one of the following commands should produce sensible output:
host clicense1
: or:
ping -c 3 clicense1
* Verify that ''both license ports on the same server'' can be reached from your client machine:
telnet '''clicense1''' 1718
telnet '''clicense1''' 1719
: or:
telnet '''clicense2''' 1718
telnet '''clicense2''' 1719
: A ''successful'' connection looks like:
Trying 146.139.x.y...
Connected to clicense1.cnm.anl.gov.
Escape character is '^]'.
:: Do not enter any data. Instead, close each connection by hitting <code>Ctrl</code><code>]</code> (closing square bracket), followed by <code>Ctrl</code><code>D</code>.
: A ''failed'' connection gives one of these errors:
telnet: connect to address 146.139.x.y: Operation timed out
: or:
telnet: connect to address 146.139.x.y: Connection refused
:: In these cases, ask that your IP firewall configuration be reviewed and changed.

=== License configuration when outside CNM and using SSH Tunneling ===
If you're comfortable with ssh and tunneling, configure client licensing as follows:

==== License format ====
* Floating Network License (FNL), "Port number and Hostname"
: '''Note:''' Three-server redundancy cannot be accessed over SSH tunneling. Instead, the user must configure just one server, and if needed in rare situations manually reconfigure to use the other server if the first one fails.

==== License server ====
* Enter into the COMSOL license file:
<source lang="make">
SERVER 127.0.0.1 ANY 1718
USE_SERVER
</source>
: '''Note:''' Do not use "localhost", as this will be directly (and incorrectly) interpreted by COMSOL to mean that the current machine actually ''is'' a license server.

Connect to <code>mega</code> or a Carbon login node with one of the following ssh commands:
* First, set up a tunnel to the primary license server:
ssh -L 1718:'''clicense1''':1718 -L 1719:'''clicense1''':1719 mega
* Start COMSOL.
: If you get an error that the number of licenses is exhausted, that is at least confirmation that you correctly reached the license server.

If the above attempt takes a long time or hangs entirely, it may be because the license service currently runs on the secondary server.
* Connect to the alternate license server:
ssh -L 1718:'''clicense2''':1718 -L 1719:'''clicense2''':1719 mega
* Start COMSOL.

== License Troubleshooting ==
The following are typical error messages.
=== Could not obtain license for COMSOL Multiphysics GUI ===
* Error details:
License error: -4.
'''Licensed number of users already reached.'''
Feature: COMSOLGUI
License path: /Users/stern/Applications/COMSOL52a/license/license.dat:
FlexNet Licensing error:-4,132
* Reason: The connection to the license server succeeded, but the license token(s) you requested are in use.
* Solution: Wait a bit, or [http://carbonmgmt/ganglia/lic.php identify who's using COMSOL] and politely ask the user when they might be finished with their work.

=== License server machine is down or not responding ===
* Error details:
License error: -96.
See the system administrator about starting the license server system, or
make sure you're referring to the right host (see LM_LICENSE_FILE).
Feature: SERIAL
Hostname: clicense3
License path: /Users/stern/Applications/COMSOL52a/license/license.dat:
FlexNet Licensing error:-96,7
* Reason: The license servers cannot be reached at all. The <code>ping</code> or <code>host</code> tests mentioned above fail.
* Solution: Review the client license file and DNS lookup domains.

=== License file does not support this version ===
* Error details:
Feature: SERIAL
'''Application version > License version: 5.21 > 3.5'''
License path: /Users/stern/Applications/COMSOL52a/license/license.dat:
FlexNet Licensing error:-21,126
* Reason: Only the first of two steps for connecting to the license servers succeeds, which may happen when the redundant license service runs on the alternate server instead of the one you're connected to.
* Solution: When using SSH, try <code>clicense2</code> instead of <code>clicense1</code>, or vice versa. Revisit your <code>~/.ssh/config</code> file or use the alternate command line as shown in the SSH section above.

== Multiple CPU cores ==
Ways to start multi-core (from http://www.comsol.com/support/knowledgebase/1096/)
comsol -np 8

export COMSOL_NUM_THREADS=8 # automatically provided on ''Carbon''
comsol
* COMSOL does not benefit from hyperthreading (oversubscribing the physical core count).

== Batch jobs ==

* http://www.comsol.com/support/knowledgebase/1001/ .
* [[Media:HPC 2012-05-16 Comsol parallel usage.pdf]] (Modified from http://www.comsol.com/support/knowledgebase/1001/files/4027/cluster_install_linux_42a.pptx – Appendix slide, PBS)
* '''Note:''' MPD is not supported on Carbon (I patched the main comsol script), and no longer necessary at all in Comsol-4.3.

=== Sample Job file ===
See:
$COMSOL_HOME/sample.job

== Important COMSOL Support Articles ==
* Toplevel suport page: https://www.comsol.com/support

=== KB Articles ===
* [https://www.comsol.com/support/knowledgebase/1001 Running COMSOL® in parallel on clusters]
* [https://www.comsol.com/support/knowledgebase/1250 Running parametric sweeps, batch sweeps, and cluster sweeps from the command line]
* [https://www.comsol.com/support/knowledgebase/1255 Reducing the amount of solution data stored in a model]
* [https://www.comsol.com/support/knowledgebase/830 Handling large models]

=== Blog-style articles ===
* [https://www.comsol.com/blogs/the-power-of-the-batch-sweep/ The Power of the Batch Sweep]
* [https://www.comsol.com/blogs/how-to-run-simulations-in-batch-mode-from-the-command-line/ How to Run Simulations in Batch Mode from the Command Line]

HPC/Applications/comsol

2025-08-07T15:06:19Z

Stern: /* Introduction */

== Introduction ==
The [http://www.comsol.com/products/multiphysics/ COMSOL Multiphysics] product is available for Mac, Windows, and Linux.
[[Image:HPC Comsol licensed modules.png|right|450px|thumb|COMSOL features licensed at CNM as of 2024-10.]]

== Access ==
=== COMSOL Multiphysics ===
CNM's license for
[http://www.comsol.com/products/multiphysics/ '''COMSOL Multiphysics''']
(the main graphical user interface application with batch computing capabilities)
is available
'''''upon request''''' 
to
'''''Argonne employees''''' 
in the following roles:
* ''CNM Facility Users'', under an active proposal with COMSOL in scope — please see [[HPC/FAQ#Access_and_proposals]].
* ''NST staff'' (including PostDocs, joint appointments, and students).

Of the available features, we have licensed a subset, shown at right.
* To have a native desktop version installed on your Argonne-owned computer (laptop or desktop), please contact [mailto:help@anl.gov?Subject=COMSOL%20license%20access%20request CNM support]. Make sure to review [http://www.comsol.com/support/updates/ Product Updates].
* To run COMSOL on Carbon itself, as GUI or as batch computing jobs, your proposal must have Carbon access in scope.

=== COMSOL Server ===
CNM Facility Users who are not Argonne employees are not eligible to access the full '''COMSOL Multiphysics''' product,
in compliance with its licensing terms and conditions.
However, the [https://www.comsol.com/comsol-server '''COMSOL Server'''] product offers an alternative but more narrow means of access.

All users, including '''''non-Argonne users''''' may be able to run, over a web interface in their browser,
pre-parameterized "fixed-model" simulations that were made with the '''COMSOL Application Builder''' (requires Windows).

To inquire about using COMSOL server applications,
please contact us.

We will review possible purchase paths for COMSOL Server, and the ''considerable effort'' likely required by users and the CNM to develop and deploy a COMSOL application.

== Support ==
* For questions on how to run or start COMSOL:
*: Submit an [[HPC/Support | '''IT support request''']]
* For questions related to most COMSOL data or scripts, contact COMSOL support at:
*: [mailto:support@comsol.com '''support'''@comsol.com]
* '''Exception:''' If your support question might contain data that is subject to '''US Export Control''' regulations, contact instead:
*: [mailto:us-export-control@us.comsol.com '''us-export-control'''@'''us'''.comsol.com]

== The modes to run COMSOL ==
There are several ways of running COMSOL.

=== Running the desktop application ===
To edit models and analyze results, use the COMSOL graphical user interface (GUI). You can run it in several ways, each explained in more detail in the following sections.

# Run on Carbon, display on your desktop using the [http://en.wikipedia.org/wiki/X_Window_System X Window system (X11)].
#* Requires an ''X11 Server'' application running on your client machine, and SSH X11 forwarding.
#* The X11 protocol can be slow over LAN.
# Run on Carbon within a [[HPC/Network Access/Virtual Desktop | virtual desktop environment (VNC)]].
#* Requires a VNC client on your machine.
#* It can be confusing where VNC and the app runs.
#* The virtual desktop presents a (deliberatly) sparse windowing environment – Copy&Paste can be difficult to use.
# Run on your own computer, remotely checking out Carbon's license.
#* Requires a full installation of COMSOL on your desktoo or laptop computer.
#* You will require access to the COMSOL license server running on Carbon, either through VPN (fairly easy) or SSH tunneling (can be intimidating for first-time users). See below for details.

=== Running a batch calculation ===
When the computational stage of models takes longer than a few minutes to complete, it will be beneficial to submit save the model file and run it in parallel on Carbon.
See the [[#Batch jobs|section below]].

== Use with X11 ==
For casual use, and with [[HPC/Network Access/SSH Tunnel Setup on Linux and MacOS | X11 tunneling to Carbon]] set up, simply add to your <code>~/.bashrc</code> file on Carbon:
module load comsol

Then, from the command line on a Carbon login node, start the COMSOL GUI:
comsol &

License configuration is automatic on Carbon.

Comsol will use a number of cores to run calculations multithreaded by default. (On Carbon, COMSOL_NUM_THREADS is set.)

'''Quit the application''' as soon as you are done. Only one user can use Comsol at a time.

== Use with VNC ==
See general [[HPC/Network Access/Virtual Desktop | instructions on using VNC]], and the COMSOL example therein.

== Native desktop version ==

To get the fastest response from the COMSOL GUI, run a native COMSOL version on your desktop machine or laptop, called ''client machine'' below.

Enter license details either in the COMSOL Setup configuration dialog.

== Native desktop version — License configuration ==
A client's COMSOL license needs to be configured depending on how the client reaches CNM network resources. Follow the appropriate section:
* [[#License configuration when inside CNM or using VPN]]
* [[#License configuration when outside CNM and using SSH Tunneling]]

=== License configuration when inside CNM or using VPN ===
==== Ensure correct DNS search domains ====

* Inspect the client machine's network configuration.
* Ensure that '''two''' entries are listed as DNS Search Domains for the current network profile:
nst.anl.gov
cnm.anl.gov
: (The order of entries does not matter for the purposes considered here.)
* If the client machine is ''wired to CNM networks'', the domains are normally provided correctly by DHCP.
* If the client machine is ''connected over VPN'', you may need edit the respective network profile.
:* On a Mac, find the VPN configuration details under System Preferences, Network, Advanced… . You may need to unlock the Network panel first before you can make changes under "Advanced…". It appears that those changes are being saved and will be activated automatically the next time that this VPN session gets started.
: [[Image:VNC_add_cnm_to_DNS_1.png|400px]] [[Image:VNC_add_cnm_to_DNS_2.png|400px]]
:* On Windows or Linux machines, take ''analogous'' steps, using the platform's own configuration tools.

* Exclude license manager components from client desktop installation. (Those are all handled server-side.)

==== License format ====
* Floating Network License (FNL), "Three-server redundancy"

==== License servers ====
* On client machines, while running the COMSOL installer or license configuration, specify license servers and ports as follows:
clicense1 1718
clicense2 1718
clicense3 1718
: '''Note:''' Use ''short host name'' form exactly as given here. (Technical Reason: The license must be usable both on Carbon's internal network as well as on divisional networks that are outside of Carbon).

'''Advanced:''' A client machine's COMSOL license file <code><COMSOLDIR>/Multiphysics/license/license.dat</code> should look as follows:
<source lang="make">
SERVER clicense1 ANY 1718
SERVER clicense2 ANY 1718
SERVER clicense3 ANY 1718
USE_SERVER
</source>

<code><COMSOLDIR></code> is located by default at:
* '''Windows''': C:\Program files\COMSOL\COMSOL''nn''
* '''Linux''': /usr/local/comsol''nn''
* '''Mac''': /Applications/COMSOL''nn''
("''nn''" is the application's abbreviated version number.)

==== Network Troubleshooting ====
When using the license access model descibed here (as opposed to tunneled over SSH), ensure the following:
* Verify that short host names can be resolved to IP addresses. Either one of the following commands should produce sensible output:
host clicense1
: or:
ping -c 3 clicense1
* Verify that ''both license ports on the same server'' can be reached from your client machine:
telnet '''clicense1''' 1718
telnet '''clicense1''' 1719
: or:
telnet '''clicense2''' 1718
telnet '''clicense2''' 1719
: A ''successful'' connection looks like:
Trying 146.139.x.y...
Connected to clicense1.cnm.anl.gov.
Escape character is '^]'.
:: Do not enter any data. Instead, close each connection by hitting <code>Ctrl</code><code>]</code> (closing square bracket), followed by <code>Ctrl</code><code>D</code>.
: A ''failed'' connection gives one of these errors:
telnet: connect to address 146.139.x.y: Operation timed out
: or:
telnet: connect to address 146.139.x.y: Connection refused
:: In these cases, ask that your IP firewall configuration be reviewed and changed.

=== License configuration when outside CNM and using SSH Tunneling ===
If you're comfortable with ssh and tunneling, configure client licensing as follows:

==== License format ====
* Floating Network License (FNL), "Port number and Hostname"
: '''Note:''' Three-server redundancy cannot be accessed over SSH tunneling. Instead, the user must configure just one server, and if needed in rare situations manually reconfigure to use the other server if the first one fails.

==== License server ====
* Enter into the COMSOL license file:
<source lang="make">
SERVER 127.0.0.1 ANY 1718
USE_SERVER
</source>
: '''Note:''' Do not use "localhost", as this will be directly (and incorrectly) interpreted by COMSOL to mean that the current machine actually ''is'' a license server.

Connect to <code>mega</code> or a Carbon login node with one of the following ssh commands:
* First, set up a tunnel to the primary license server:
ssh -L 1718:'''clicense1''':1718 -L 1719:'''clicense1''':1719 mega
* Start COMSOL.
: If you get an error that the number of licenses is exhausted, that is at least confirmation that you correctly reached the license server.

If the above attempt takes a long time or hangs entirely, it may be because the license service currently runs on the secondary server.
* Connect to the alternate license server:
ssh -L 1718:'''clicense2''':1718 -L 1719:'''clicense2''':1719 mega
* Start COMSOL.

== License Troubleshooting ==
The following are typical error messages.
=== Could not obtain license for COMSOL Multiphysics GUI ===
* Error details:
License error: -4.
'''Licensed number of users already reached.'''
Feature: COMSOLGUI
License path: /Users/stern/Applications/COMSOL52a/license/license.dat:
FlexNet Licensing error:-4,132
* Reason: The connection to the license server succeeded, but the license token(s) you requested are in use.
* Solution: Wait a bit, or [http://carbonmgmt/ganglia/lic.php identify who's using COMSOL] and politely ask the user when they might be finished with their work.

=== License server machine is down or not responding ===
* Error details:
License error: -96.
See the system administrator about starting the license server system, or
make sure you're referring to the right host (see LM_LICENSE_FILE).
Feature: SERIAL
Hostname: clicense3
License path: /Users/stern/Applications/COMSOL52a/license/license.dat:
FlexNet Licensing error:-96,7
* Reason: The license servers cannot be reached at all. The <code>ping</code> or <code>host</code> tests mentioned above fail.
* Solution: Review the client license file and DNS lookup domains.

=== License file does not support this version ===
* Error details:
Feature: SERIAL
'''Application version > License version: 5.21 > 3.5'''
License path: /Users/stern/Applications/COMSOL52a/license/license.dat:
FlexNet Licensing error:-21,126
* Reason: Only the first of two steps for connecting to the license servers succeeds, which may happen when the redundant license service runs on the alternate server instead of the one you're connected to.
* Solution: When using SSH, try <code>clicense2</code> instead of <code>clicense1</code>, or vice versa. Revisit your <code>~/.ssh/config</code> file or use the alternate command line as shown in the SSH section above.

== Multiple CPU cores ==
Ways to start multi-core (from http://www.comsol.com/support/knowledgebase/1096/)
comsol -np 8

export COMSOL_NUM_THREADS=8 # automatically provided on ''Carbon''
comsol
* COMSOL does not benefit from hyperthreading (oversubscribing the physical core count).

== Batch jobs ==

* http://www.comsol.com/support/knowledgebase/1001/ .
* [[Media:HPC 2012-05-16 Comsol parallel usage.pdf]] (Modified from http://www.comsol.com/support/knowledgebase/1001/files/4027/cluster_install_linux_42a.pptx – Appendix slide, PBS)
* '''Note:''' MPD is not supported on Carbon (I patched the main comsol script), and no longer necessary at all in Comsol-4.3.

=== Sample Job file ===
See:
$COMSOL_HOME/sample.job

== Important COMSOL Support Articles ==
* Toplevel suport page: https://www.comsol.com/support

=== KB Articles ===
* [https://www.comsol.com/support/knowledgebase/1001 Running COMSOL® in parallel on clusters]
* [https://www.comsol.com/support/knowledgebase/1250 Running parametric sweeps, batch sweeps, and cluster sweeps from the command line]
* [https://www.comsol.com/support/knowledgebase/1255 Reducing the amount of solution data stored in a model]
* [https://www.comsol.com/support/knowledgebase/830 Handling large models]

=== Blog-style articles ===
* [https://www.comsol.com/blogs/the-power-of-the-batch-sweep/ The Power of the Batch Sweep]
* [https://www.comsol.com/blogs/how-to-run-simulations-in-batch-mode-from-the-command-line/ How to Run Simulations in Batch Mode from the Command Line]