Job Processing on the Linux-Cluster

Page Content

This document describes how to use the CoolMUC-4, i.e., how to find the appropriate compute resource, how to setup serial, parallel or interactive compute jobs and how to start and manage the jobs on the cluster. Basic job examples are provided.

What you should know at the beginning

All programs in the parallel or serial segments of the cluster must be started up using either

  • a SLURM batch script or
  • an interactive SLURM shell.

In order to access the SLURM infrastructure described here, please first log in to a login node of the cluster as described in Access and Login to the Linux-Cluster.

This document and its subdocuments provide information on:

Choose the compute resources of the Linux Cluster with great care! In particular, please be aware that misuse (see below) of the resources described here can result in the invalidation of the violating account!

Do you need help or do you fear misusing the HPC resources?

Just get in touch with us for further consulting via Linux Cluster request at LRZ Servicedesk or via HPC Lounge.

Step 1: Get the resource which fits my needs aka "jobs that the Slurm scheduler likes"

In order to allocate the optimal resource for your job, you may consider different job types, e.g.:

  • standard distributed memory parallel job, e.g. using MPI
  • shared-memory parallel job on a single node, e.g. using OpenMP
  • single-core (serial) job
  • single-node parallel job requiring a fraction of both the available CPU cores and memory
  • single-node parallel job requiring a fraction of the available CPU cores but needing almost all memory of a regular compute node on CoolMUC-4
  • large-memory job

The following decision matrix provides assistance in making the right choice.

Decision matrix to select the appropriate partition for
production jobs (interactive tests)
How many CPU resources
does my job need?
How much memory does my job need?
up to 488 GB
per node
489 - 1000 GB
per node
up to 6000 GB
per node
1 - 10 CPU cores

serial_std
serial_long
(cm4_inter)

serial_std
serial_long
(cm4_inter, teramem_inter)

teramem_inter
(teramem_inter)

more than 8 CPU cores
and max. 1 compute node

cm4_tiny
(cm4_inter)
teramem_interteramem_inter

2 - 4 compute nodes

cm4_std
(cm4_inter)

 

 
more than 4 compute nodes

CoolMUC-4 cannot satisfy this requirement!
Please check whether an application for a (test) project on SuperMUC-NG is a suitable alternative.

Step 2: Cluster Overview

Slurm
cluster
segment
Slurm
partition

Compute
nodes
in partition

CPU cores
per node

GPUs
per node

Memory
per node

Cluster system CoolMUC-4: Sapphire Rapids (Intel(R) Xeon(R) Platinum 8480+) nodes
cm4cm4_std

100

(overlapping
partitions)

112 (physical)

224 (logical)

--

488 GiB

cm4_tiny
inter
cm4_inter6--488 GiB
Cluster system CoolMUC-4: Ice Lake (Intel(R) Xeon(R) Platinum 8380) nodes
serialserial_std10

80 (physical)

160 (logical)

--

1000 GiB

serial_long

2

--

Cluster system Teramemsingle-node shared-memory system (Intel Xeon Platinum 8360HL)

interteramem_inter1

96 (physical)

192 (logical)

--5.9 TiB

Step 3: What job specifications do I have to set and what limits do I have to consider?

Check the job limits of the cluster partitions and set the appropriate specifications for your jobs. For full job script examples, please consult the documents the following table is referring to.

Slurm
partition

Node range
per job
min - max

CPU range
per job
(physical cores)
min - max

Maximum
job runtime
(hours)

Maximum
running (submitted)
jobs
per user

Memory limit

Node
usage
Mandatory
job specifications
Remarks
cm4_std2 - 4

112 - 448

242 (25)

488 GiB per node

exclusive
--clusters=cm4
--partition=cm4_std
--qos=cm4_std
Running parallel jobs on the Linux Cluster
cm4_tiny1 - 1

9 - 112

244 (25)

default: 2.1 GiB per logical CPU core

overall limit: 488 GiB per node

shared
--clusters=cm4
--partition=cm4_tiny
cm4_inter1 - 4

1 - 112

8

1 (2)

--clusters=inter
--partition=cm4_inter
Running interactive jobs on the Linux Cluster
serial_std1 - 1
1 - 1624

96 (200)

default: 6.2 GiB per logical CPU core

overall limit: 1000 GiB per node

shared
--clusters=serial
--partition=serial_std

Additional limit: Number of CPU cores in sum over all running jobs is limited to 96!

Running serial jobs on the Linux Cluster

serial_long168

96 (200)

--clusters=serial
--partition=serial_long
teramem_inter

1 - 1

1 - 96240

1 (1)

approx. 60 GiB
per physical core
available

shared

--clusters=inter
--partition=teramem_inter
Running large-memory jobs on the Linux Cluster


Remarks on CPU limit

CPU limit refers to the number of available hardware (physical) cores. When setting up your Slurm jobs, please consider the following characteristics and restrictions:

  • Jobs may incorporate all logical cores, i.e., the number of physical cores times number of threads that can simultaneously run on one core via hyperthreading. In case of CoolMUC-4, this is twice the number of physical cores. Shared-memory jobs (e.g. using OpenMP) or hybrid jobs (e.g. using MPI + OpenMP) may benefit from that. Please also refer to our Slurm job examples.
  • The total number of requested cores, i.e. the product of "tasks per node" (e.g. MPI processes) and "CPUs per tasks" (e.g. OpenMP threads) ...
    • Must not exceed the maximum number of cores which is available or allowed to be used per compute node on a particular partition!
    • Must not be smaller than the minimum number of cores allowed to be used on a compute node!
  • Important: Due to limited hardware resources in the serial cluster, the following restrictions apply:
    • Only a subset of CPU cores per node can be used per job (see table)! 
    • The total number of requested CPU cores by a user on the entire serial cluster, which are in use at the same time in running jobs, is also limited (see table)! You may distribute the CPU cores to different jobs.

The terms "tasks per node" and "CPUs per tasks" refer to the according Slurm specifications "--tasks-per-node" and "--cpus-per-task". Please refer to our Slurm documentation and CoolMUC-4 job script examples.

Remarks on Memory limit

Keep in mind that the default memory per job on shared partitions (serial cluster, cm4_tiny, cm4_inter) scales with the number of allocated CPU cores. In other words, by default, your job will only get a portion of the memory, according to the number of requested CPU cores!
If more memory is required, this has to be defined in the job via the "--mem" option.

General remarks

  • Nodes on partitions cm4_tiny, cm4_interserial_stdserial_long and teramem_inter are used as shared resources, i.e., multiple jobs/users share those nodes. cm4_std only provides exclusive nodes for jobs.
  • Partitions cm4_std, cm4_tiny, serial_std and serial_long are only intended for batch jobs (via sbatch command, see below)!
  • The partition cm4_inter is intended for interactive jobs (via salloc command, see below). Due to the short job runtime, this partition is suitable for small test jobs.
  • Both batch jobs and interactive jobs can be run on the partition teramem_inter.

Common Slurm commands on the Linux Cluster for job submission and job management

Once submitted, a Slurm job will be queued for some time, depending on how many jobs are presently submitted and how many resources are available. Eventually, more or less after previously submitted jobs have completed, the job will be started on one or more of the systems determined by its resource requirements. Slurm provides several commands to check the status of waiting or running jobs, to inspect or even modify (modification only to limited extent) waiting/running jobs, to obtain information of finished jobs and to delete waiting/running jobs. Click here to get an overview of commonly used Slurm commands on the Linux Cluster.