General

Apptainer (formerly Singularity before its open-source version was transferred to the Linux Foundation) is a container virtualization platform specifically designed for HPC environments to run complex applications on HPC clusters in a simple, portable, and reproducible way. Containers can be seen as lightweight virtual operating systems with preinstalled and preconfigured software that can be run just like any other program on the cluster. Apptainer most importantly focuses on:

• Verifiable reproducibility and security, using cryptographic signatures, an immutable container image format, and in-memory decryption.
• Integration over isolation by default. Easily make use of GPUs, high speed networks, parallel filesystems on a cluster or server by default.
• Mobility of compute. The single file SIF container format is easy to transport and share.
• A simple, effective security model. You are the same user inside a container as outside, and cannot gain additional privilege on the host system by default.

Modules providing Apptainer

module av apptainer

module load apptainer

The Apptainer modules automatically set two important environment variables:

Command line (CLI) autocompletion is working as well (type apptainer in the terminal, then press the TAB key).

Apptainer Basics

Apptainer Documentation

Apptainer has a very rich and extensive online documentation as well as man pages and CLI documentation:

$ apptainer help

Linux container platform optimized for High Performance Computing (HPC) and
Enterprise Performance Computing (EPC)

Usage:
  apptainer [global options...]

Description:
  Apptainer containers provide an application virtualization layer enabling
  mobility of compute via both application and environment portability. With
  Apptainer one is capable of building a root file system that runs on any
  other Linux system where Apptainer is installed.

Options:
      --build-config    use configuration needed for building containers
  -c, --config string   specify a configuration file (for root or
                        unprivileged installation only) (default
                        "/hpc/gpfs2/sw/apptainer/1.3.4/etc/apptainer/apptainer.conf")
  -d, --debug           print debugging information (highest verbosity)
  -h, --help            help for apptainer
      --nocolor         print without color output (default False)
  -q, --quiet           suppress normal output
  -s, --silent          only print errors
  -v, --verbose         print additional information
      --version         version for apptainer

Available Commands:
  build       Build an Apptainer image
  cache       Manage the local cache
  capability  Manage Linux capabilities for users and groups
  checkpoint  Manage container checkpoint state (experimental)
  completion  Generate the autocompletion script for the specified shell
  config      Manage various apptainer configuration (root user only)
  delete      Deletes requested image from the library
  exec        Run a command within a container
  help        Help about any command
  inspect     Show metadata for an image
  instance    Manage containers running as services
  key         Manage OpenPGP keys
  keyserver   Manage apptainer keyservers
  oci         Manage OCI containers
  overlay     Manage an EXT3 writable overlay image
  plugin      Manage Apptainer plugins
  pull        Pull an image from a URI
  push        Upload image to the provided URI
  registry    Manage authentication to OCI/Docker registries
  remote      Manage apptainer remote endpoints
  run         Run the user-defined default command within a container
  run-help    Show the user-defined help for an image
  search      Search a Container Library for images
  shell       Run a shell within a container
  sif         Manipulate Singularity Image Format (SIF) images
  sign        Add digital signature(s) to an image
  test        Run the user-defined tests within a container
  verify      Verify digital signature(s) within an image
  version     Show the version for Apptainer

Examples:
  $ apptainer help <command> [<subcommand>]
  $ apptainer help build
  $ apptainer help instance start


For additional help or support, please visit https://apptainer.org/help

Pulling Containers

You can use the pull command to download images from an external resource like an OCI registry and save them to a sif-file.

You can use pull with the docker:// uri to reference OCI images served from an OCI registry. In this case pull does not just download an image file. OCI images are stored in layers, so pull must also combine those layers into a usable Apptainer sif-file. Both layers and Apptainer sif-files will be cached in $APPTAINER_CACHEDIR.

apptainer pull docker://archlinux
apptainer pull docker://sylabsio/lolcow
apptainer pull docker://quay.io/bitnami/python:3.12
apptainer pull docker://nvcr.io/nvidia/pytorch:21.09-py3
apptainer pull docker://ghcr.io/containerd/alpine:latest

Running Containers

There are three ways to run a container, mainly differing in what will be executed in the container:

apptainer shell archlinux.sif

apptainer exec archlinux.sif bash script.bash

apptainer run archlinux.sif

apptainer run docker://archlinux:latest

  -B, --bind stringArray              a user-bind path specification.
                                      spec has the format
                                      src[:dest[:opts]], where src and
                                      dest are outside and inside paths.
                                      If dest is not given, it is set
                                      equal to src.  Mount options
                                      ('opts') may be specified as 'ro'
                                      (read-only) or 'rw' (read/write,
                                      which is the default). Multiple bind
                                      paths can be given by a comma
                                      separated list.
  -e, --cleanenv                      clean environment before running
                                      container
      --env stringToString            pass environment variable to
                                      contained process (default [])
      --env-file string               pass environment variables from file
                                      to contained process
  -f, --fakeroot                      run container with the appearance of
                                      running as root
  -c, --contain                       use minimal /dev and empty other
                                      directories (e.g. /tmp and $HOME)
                                      instead of sharing filesystems from
                                      your host
  -C, --containall                    contain not only file systems, but
                                      also PID, IPC, and environment
      --no-eval                       do not shell evaluate env vars or
                                      OCI container CMD/ENTRYPOINT/ARGS
      --no-home                       do NOT mount users home directory if
                                      /home is not the current working
                                      directory
      --no-init                       do NOT start shim process with --pid
      --no-mount strings              disable one or more 'mount xxx'
                                      options set in apptainer.conf and/or
                                      specify absolute destination path to
                                      disable a bind path entry, or
                                      'bind-paths' to disable all bind
                                      path entries.
  -W, --workdir string                working directory to be used for
                                      /tmp, /var/tmp and $HOME (if
                                      -c/--contain was also used)
      --nv                            enable Nvidia support
      --nvccli                        use nvidia-container-cli for GPU
                                      setup (experimental)
  -w, --writable                      by default all Apptainer containers
                                      are available as read only. This
                                      option makes the file system
                                      accessible as read/write.
      --writable-tmpfs                makes the file system accessible as
                                      read-write with non persistent data
                                      (with overlay support only)

Example Slurm Job Template for Running Containers

#!/usr/bin/env bash
 
#SBATCH --job-name=test
#SBATCH --partition=epyc
#SBATCH --mail-type=END,INVALID_DEPEND
#SBATCH --mail-user=noreply@uni-augsburg.de
#SBATCH --time=1-0
 
# Request memory per CPU
#SBATCH --mem-per-cpu=1G
# Request n CPUs for your task.
#SBATCH --cpus-per-task=n
 
# set number of OpenMP threads
export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK
 
module purge
module load apptainer

srun apptainer ...

Passing Environment Variables to Containers

By default, most environment variables are propagated to the container by default. Exceptions are:

1. An environment variable set on the host will be overridden by a variable of the same name that has been set either inside the container image, or via APPTAINERENV_ environment variables, or the --env and --env-file flags.
2. The PS1 shell prompt is reset for a container specific prompt.
3. The PATH environment variable will be modified to contain default values.
4. The LD_LIBRARY_PATH is modified to a default /.singularity.d/libs, that will include NVIDIA / ROCm libraries if applicable.

To override an environment variable that is already set in the container with the value from the host, use:

export APPTAINERENV_MYVAR="$MYVAR"
apptainer run mycontainer.sif

apptainer run --env "MYVAR=$MYVAR" mycontainer.sif

cat > file <<EOF
MYVAR="$MYVAR"
EOF

apptainer run --env-file file mycontainer.sif

If you set a variable on your host called APPTAINERENV_PREPEND_PATH then its value will be prepended (added to the start) of the PATH variable in the container.

$ apptainer exec mycontainer.sif sh -c 'echo $PATH'
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

$ export APPTAINERENV_PREPEND_PATH="/startpath"
$ apptainer exec mycontainer.sif sh -c 'echo $PATH'
/startpath:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

# or
$ apptainer exec --env PREPEND_PATH=/startpath mycontainer.sif sh -c 'echo $PATH'
/startpath:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

For a more detailed explanation for environment variable precedence see Apptainer documentation.

Using Containers exported from docker

If you have a customized docker container on your workstation, you may export it there and convert it to a sif-container on the cluster:

# workstation
docker images # show all docker containers
docker save $IMAGEID -o container.tar
# HPC cluster
apptainer build container.sif docker-archive:container.tar

Inspecting Container Metadata

apptainer inspect archlinux.sif
apptainer inspect --json archlinux.sif
apptainer inspect --all archlinux.sif

Best Practices using Apptainer

Don't save container images in your $HOME

Container images can become quite big in size and can typically be recreated easily. All your data in $HOME is backed up to tape every day, so it should be avoided to store container images as well as non-essential or temporary data there.

Use SIF containers exclusively

SIF container files are the native Apptainer container format and provide the best performance. Avoid running directory based (sandbox) containers productively.

Unload unnecessary Lmod modules

Most of the available modules on the cluster won't work in a container (MPI modules may be an exception). Since the users environment is automatically propagated to the container. This may break a container (especially when compiler modules are loaded). In order to avoid such issues, use ml purge in your Job scripts before loading apptainer, or use the --contain-env CLI option to disable environment propagation.

Differences and Limitations vs Docker

Read-only by Default

Apptainer’s container image format (SIF) is generally read-only. This permits containers to be run in parallel from a shared location on a network filesystem, support in-built signing and verification, and offer encryption. A container’s filesystem is mounted directly from the SIF, as SquashFS, so cannot be written to by default.

When a container is run using Docker its layers are extracted, and the resulting container filesystem can be written to and modified by default. If a Docker container expects to write files, you will need to follow one of the following methods to allow it to run under Apptainer.

• A directory from the host can be passed into the container with the --bind or --mount flags. It needs to be mounted inside the container at the location where files will be written.

• The --writable-tmpfs flag can be used to allow files to be created in a special temporary overlay. Each time the container is executed, a separate temporary overlay is used and then discarded when the container exits. The SIF Container image file is never modified.

• The container can be converted to a sandbox directory, and executed with the --writable flag, which allows modification of the sandbox content.

• A writable overlay partition can be added to the SIF file, and the container executed with the --writable flag. Any changes made are kept permanently in the overlay partition.

Default Mounts / $HOME

A default installation of Apptainer will mount the user’s home directory, /tmp directory, and the current working directory, into each container that is run. Docker does not mount host directories into the container by default.

The home directory mount is the most likely to cause problems when running Docker containers. Various software will look for packages, plugins, and configuration files in $HOME. If you have, for example, installed packages for Python into your home directory (pip install --user) then a Python container may find and attempt to use them. This can cause conflicts and unexpected behavior.

If you experience issues, use the --contain option to stop Apptainer automatically binding directories into the container. You may need to use --bind or --mount to then add back e.g. an HPC project directory that you need access to.

Environment Propagation

Apptainer propagates most environment variables set on the host into the container, by default. Docker does not propagate any host environment variables into the container. Environment variables may change the behaviour of software.

To disable automatic propagation of environment variables, the --cleanenv / -e flag can be specified. When --cleanenv is used, only variables on the host that are prefixed with APPTAINERENV_ are set in the container.

Any environment variables set via an ENV line in a Dockerfile will be available when the container is run with Apptainer. You can override them with APPTAINERENV_ vars, or the --env / --env-file flags, but they will not be overridden by host environment variables.

CMD / ENTRYPOINT Behavior

When a container is run using docker, its default behavior depends on the CMD and/or ENTRYPOINT set in the Dockerfile that was used to build it, along with any arguments on the command line. The CMD and ENTRYPOINT can also be overridden by flags.

An Apptainer container has the concept of a runscript, which is a single shell script defining what happens when you apptainer run the container. Because there is no internal concept of CMD and ENTRYPOINT, Apptainer must create a runscript from the CMD and ENTRYPOINT when converting a Docker container. The behavior of this script mirrors Docker as closely as possible:

# ENTRYPOINT="date"

# Runs 'date'
$ apptainer run mycontainer.sif
Wed 06 Oct 2021 02:42:54 PM CDT

# Runs 'date --utc`
$ apptainer run mycontainer.sif --utc
Wed 06 Oct 2021 07:44:27 PM UTC

# CMD="date"

# Runs 'date'
$ apptainer run mycontainer.sif
Wed 06 Oct 2021 02:45:39 PM CDT

# Runs 'echo hello'
$ apptainer run mycontainer.sif echo hello
hello

# ENTRYPOINT="date"
# CMD="--utc"

# Runs 'date --utc'
$ apptainer run mycontainer.sif
Wed 06 Oct 2021 07:48:43 PM UTC

# Runs 'date -R'
$ apptainer run mycontainer.sif -R
Wed, 06 Oct 2021 14:49:07 -0500

Best Practices for Docker & Apptainer Compatibility

As detailed previously, Apptainer can make use of most Docker and OCI images without issues, or via simple workarounds. In general, however, there are some best practices that should be applied when creating Docker / OCI containers that will also be run using Apptainer.

Don’t require execution by a specific user

Avoid using the USER instruction in your Docker file, as it is ignored by Apptainer. Install and configure software inside the container so that it can be run by any user.

Don’t install software under /root or in another user’s home directory

Because a Docker container builds and runs as the root user by default, it’s tempting to install software into root’s home directory (/root). Permissions on /root are usually set so that it is inaccessible to non-root users. When the container is run as another user the software may be inaccessible.

Software inside another user’s home directory, e.g. /home/myapp, may be obscured by Apptainer’s automatic mounts onto /home.

Install software into system-wide locations in the container, such as under /usr or /opt to avoid these issues.

Support a read-only filesystem

Because of the immutable nature of the SIF format, a container run with Apptainer is read-only by default.

Try to ensure your container will run with a read-only filesystem. If this is not possible, document exactly where the container needs to write, so that a user can bind in a writable location, or use --writable-tmpfs as appropriate.

You can test read-only execution with Docker using docker run --read-only --tmpfs /run --tmpfs /tmp container.

Be careful writing to /tmp

Apptainer mounts the host /tmp into the container, by default. This means you must be be careful when writing sensitive information to /tmp, and should ensure your container cleans up files it writes there.

Consider library caches / ldconfig

If your Dockerfile adds libraries and / or manipulates the ld search path in the container (ld.so.conf / ld.so.conf.d), you should ensure the library cache is updated during the build.

Because Apptainer runs containers read-only by default, the cache and any missing library symlinks may not be able to be updated / created at execution time.

Run ldconfig toward the end of your Dockerfile to ensure symbolic links and the the ld.so.cache are up-to-date.

Building containers

The build command is the “Swiss army knife” of container creation. You can use it to download and assemble existing containers from external resources like Docker Hub and other OCI registries. You can use it to convert containers between the formats supported by Apptainer. And you can use it in conjunction with a Apptainer definition file to create a container from scratch and customized it to fit your needs.

Using a definition file

A definition file is the Apptainer equivalent of a Dockerfile and basically allows you to build customized containers based on imported containers.

Bootstrap: docker
From: archlinux:latest

%files
    testfile /opt

%post
	pacman -Sy
    pacman -S --noconfirm cowsay

%environment
    export PATH=/usr/games:$PATH

%runscript
    date | cowsay

apptainer build archlinux.sif archlinux.def

Interactively

If you want to create a container with a writable directory (called a sandbox) you can do so with the --sandbox option.

apptainer build --sandbox archlinux/ docker://archlinux:latest

apptainer shell --writable archlinux/

apptainer build archlinux.sif archlinux/

apptainer build --sandbox archlinux archlinux.sif

Using GPUs inside containers

To make use of GPUs, simply pass the --nv flag.

apptainer exec --nv ollama.sif ollama serve

WARNING: Could not find any nv files on this host! 

Using Container instances

Interacting with containers running in the background

Apptainer is most used to run containers interactively, or in a batch job, where the container runs in the foreground (using run, exec and shell), performs some work, and then exits. Apptainer, also allows you to run containers in the background in a “detached” or “daemon” mode where the container runs a service. An Apptainer container running a service in the background is called an instance, to distinguish it from the default mode which runs containers in the foreground.

apptainer instance start mycontainer.sif myinstance1

This command causes Apptainer to create an isolated environment for the container services to live inside (but will still inherit all environment variables present). It will execute the contents of the startscript file which can be defined when you build the container via the def file. You can also use the instance run command if you want the container to execute the runscript when the instance initiates.

You may run multiple instances of the same container if there is no conflict in open ports.

apptainer instance list
apptainer instance list --logs # show locations of container logfiles

apptainer exec instance://myinstance1 cat /etc/os-release
apptainer run instance://myinstance1 # run the %runscript
apptainer shell instance://myinstance1 # open a shell on an instance

apptainer instance stop myinstance1 # stops only myinstance1
apptainer instance stop myinstance* # stops all instances starting with myinstance
apptainer instance stop --all 		# stops all instances
apptainer instance stop -a    		# stops all instances

Furthermore, in contrast to Docker, Apptainer has the concept of so-called apps, which allow for multiple (different) startscripts per container that can be called using the --app CLI option.

apptainer instance start --app myapp mycontainer.sif myapp-instance

When this app instance is started, it will begin to run the %myappstart from the container’s definition file in the background. If there is no %myappstart the container will stay idle in the background.

Example Slurm Job Template for Running Container Instances

#!/usr/bin/env bash
 
#SBATCH --job-name=test
#SBATCH --partition=epyc
#SBATCH --mail-type=END,INVALID_DEPEND
#SBATCH --mail-user=noreply@uni-augsburg.de
#SBATCH --time=1-0
 
# Request memory per CPU
#SBATCH --mem-per-cpu=1G
# Request n CPUs for your task. At least 2 (1 for container, 1 for script)
#SBATCH --cpus-per-task=n 
 
# set number of OpenMP threads to n/2 to allow both the container
# as well as the python script to utilize n/2 cpu cores.
export OMP_NUM_THREADS=$((SLURM_CPUS_PER_TASK/2))
 
module purge
module load apptainer

apptainer instance start mycontainer.sif myinstance1

module load anaconda
conda activate env

srun python3 ...

Using MPI with containers

The usage of MPI applications works in principle, however the documentation for this cluster is currently a Work-in-Progress.

Cache Maintenance

When working with a lot of different container images the size of the cache folder in your personal scratch space may become huge. Use apptainer cache clean which allows for the following options:

-D, --days int       remove all cache entries older than specified number of days
-n, --dry-run        operate in dry run mode and do not actually clean the cache
-f, --force          suppress any prompts and clean the cache
-h, --help           help for clean
-T, --type strings   a list of cache types to clean (possible values: library, oci, shub, blob, net, oras, all) (default [all])

apptainer cache clean --days 30