Job Processing on the Linux-Cluster
This document describes how to use the CoolMUC-4, i.e., how to find the appropriate compute resource, how to setup serial, parallel or interactive compute jobs and how to start and manage the jobs on the cluster. Basic job examples are provided.
What you should know at the beginning
All programs in the parallel or serial segments of the cluster must be started up using either
- a SLURM batch script or
- an interactive SLURM shell.
In order to access the SLURM infrastructure described here, please first log in to a login node of the cluster as described in Access and Login to the Linux-Cluster.
This document and its subdocuments provide information on:
- Linux Cluster segments and their specifications,
- resource limits imposed for various job classes, comprising runtime limits, limits on CPU resources, memory limits and job submission limits
- how to setup and manage Slurm jobs:
Choose the compute resources of the Linux Cluster with great care! In particular, please be aware that misuse (see below) of the resources described here can result in the invalidation of the violating account!
Do you need help or do you fear misusing the HPC resources?
Just get in touch with us for further consulting via Linux Cluster request at LRZ Servicedesk or via HPC Lounge.
Step 1: Get the resource which fits my needs aka "jobs that the Slurm scheduler likes"
In order to allocate the optimal resource for your job, you may consider different job types, e.g.:
- standard distributed memory parallel job, e.g. using MPI
- shared-memory parallel job on a single node, e.g. using OpenMP
- single-core (serial) job
- single-node parallel job requiring a fraction of both the available CPU cores and memory
- single-node parallel job requiring a fraction of the available CPU cores but needing almost all memory of a regular compute node on CoolMUC-4
- large-memory job
The following decision matrix provides assistance in making the right choice.
| Decision matrix to select the appropriate partition for production jobs (interactive tests) | |||
|---|---|---|---|
| How many CPU resources does my job need? | How much memory does my job need? | ||
| up to 488 GB per node | 489 - 1000 GB per node | up to 6000 GB per node | |
| 1 - 10 CPU cores | serial_std | serial_std | teramem_inter (teramem_inter) |
more than 8 CPU cores | cm4_tiny (cm4_inter) | teramem_inter | teramem_inter |
2 - 4 compute nodes | cm4_std |
| |
| more than 4 compute nodes | CoolMUC-4 cannot satisfy this requirement! | ||
Step 2: Cluster Overview
| Slurm cluster segment | Slurm partition | Compute | CPU cores | GPUs | Memory |
| Cluster system CoolMUC-4: Sapphire Rapids (Intel(R) Xeon(R) Platinum 8480+) nodes | |||||
| cm4 | cm4_std | 100 (overlapping | 112 (physical) 224 (logical) | -- | 488 GiB |
| cm4_tiny | |||||
| inter | cm4_inter | 6 | -- | 488 GiB | |
| Cluster system CoolMUC-4: Ice Lake (Intel(R) Xeon(R) Platinum 8380) nodes | |||||
| serial | serial_std | 10 | 80 (physical) 160 (logical) | -- | 1000 GiB |
| serial_long | 2 | -- | |||
Cluster system Teramem: single-node shared-memory system (Intel Xeon Platinum 8360HL) | |||||
| inter | teramem_inter | 1 | 96 (physical) 192 (logical) | -- | 5.9 TiB |
Step 3: What job specifications do I have to set and what limits do I have to consider?
Check the job limits of the cluster partitions and set the appropriate specifications for your jobs. For full job script examples, please consult the documents the following table is referring to.
| Slurm partition | Node range | CPU range | Maximum | Maximum running (submitted) jobs per user | Memory limit | Node usage | Mandatory job specifications | Remarks |
|---|---|---|---|---|---|---|---|---|
| cm4_std | 2 - 4 | 112 - 448 | 24 | 2 (25) | 488 GiB per node | exclusive | --clusters=cm4 | Running parallel jobs on the Linux Cluster |
| cm4_tiny | 1 - 1 | 9 - 112 | 24 | 4 (25) | default: 2.1 GiB per logical CPU core overall limit: 488 GiB per node | shared | --clusters=cm4 | |
| cm4_inter | 1 - 4 | 1 - 112 | 8 | 1 (2) | --clusters=inter | Running interactive jobs on the Linux Cluster | ||
| serial_std | 1 - 1 | 1 - 16 | 24 | 96 (200) | default: 6.2 GiB per logical CPU core overall limit: 1000 GiB per node | shared | --clusters=serial | Additional limit: Number of CPU cores in sum over all running jobs is limited to 96! |
| serial_long | 168 | 96 (200) | --clusters=serial | |||||
| teramem_inter | 1 - 1 | 1 - 96 | 240 | 1 (1) | approx. 60 GiB | shared | --clusters=inter | Running large-memory jobs on the Linux Cluster |
Remarks on CPU limit
CPU limit refers to the number of available hardware (physical) cores. When setting up your Slurm jobs, please consider the following characteristics and restrictions:
- Jobs may incorporate all logical cores, i.e., the number of physical cores times number of threads that can simultaneously run on one core via hyperthreading. In case of CoolMUC-4, this is twice the number of physical cores. Shared-memory jobs (e.g. using OpenMP) or hybrid jobs (e.g. using MPI + OpenMP) may benefit from that. Please also refer to our Slurm job examples.
- The total number of requested cores, i.e. the product of "tasks per node" (e.g. MPI processes) and "CPUs per tasks" (e.g. OpenMP threads) ...
- Must not exceed the maximum number of cores which is available or allowed to be used per compute node on a particular partition!
- Must not be smaller than the minimum number of cores allowed to be used on a compute node!
- Important: Due to limited hardware resources in the serial cluster, the following restrictions apply:
- Only a subset of CPU cores per node can be used per job (see table)!
- The total number of requested CPU cores by a user on the entire serial cluster, which are in use at the same time in running jobs, is also limited (see table)! You may distribute the CPU cores to different jobs.
The terms "tasks per node" and "CPUs per tasks" refer to the according Slurm specifications "--tasks-per-node" and "--cpus-per-task". Please refer to our Slurm documentation and CoolMUC-4 job script examples.
Remarks on Memory limit
Keep in mind that the default memory per job on shared partitions (serial cluster, cm4_tiny, cm4_inter) scales with the number of allocated CPU cores. In other words, by default, your job will only get a portion of the memory, according to the number of requested CPU cores!
If more memory is required, this has to be defined in the job via the "--mem" option.
General remarks
- Nodes on partitions cm4_tiny, cm4_inter, serial_std, serial_long and teramem_inter are used as shared resources, i.e., multiple jobs/users share those nodes. cm4_std only provides exclusive nodes for jobs.
- Partitions cm4_std, cm4_tiny, serial_std and serial_long are only intended for batch jobs (via sbatch command, see below)!
- The partition cm4_inter is intended for interactive jobs (via salloc command, see below). Due to the short job runtime, this partition is suitable for small test jobs.
- Both batch jobs and interactive jobs can be run on the partition teramem_inter.
Common Slurm commands on the Linux Cluster for job submission and job management
Once submitted, a Slurm job will be queued for some time, depending on how many jobs are presently submitted and how many resources are available. Eventually, more or less after previously submitted jobs have completed, the job will be started on one or more of the systems determined by its resource requirements. Slurm provides several commands to check the status of waiting or running jobs, to inspect or even modify (modification only to limited extent) waiting/running jobs, to obtain information of finished jobs and to delete waiting/running jobs. Click here to get an overview of commonly used Slurm commands on the Linux Cluster.