Building software in user space with spack
This page give you a short introduction how to use spack at LRZ based on a preconfigured installation. It make use of the LRZ provided software stack. This documentation focuses on the LRZ specific configuration. If you haven't used spack before, it is probably advisable to go through a tutorial https://spack-tutorial.readthedocs.io/en/latest/
Note that details in the tutorial may differ due to different versions of spack.
Activate Spack
To get started with user space spack just do a
> module load user_spack
which loads a setup script that adds the spack function to your shell. Currently, we only support bash-like shells. The spack version matches the version the default software stack has been built. If you need the user spack for a different version, you can check the availability with `module av user_spack`.
Version and Documentation
To see if everything works correctly try a
> spack -V 0.13.4
and obtain the spack release version in use. LRZ will typically stick to a release version of spack for each software stack, shown here for the release stack 20.1. If you haven't used spack before, you may want to check out the documentation at
Make sure that you select the documentation for the spack version in use, since the software evolves quickly and new features may not be available in the version used for the software stack.
Basic Spack Commands
Here are the most important commands to find your way around in the spack installation.
spack list
To check if a package is available in spack you can use the list command
> spack list kokk ==> 2 packages. kokkos kokkos-kernels
So two packages match the search string.
spack info
Details are given by the info command
> spack info kokkos Package: kokkos Description: Kokkos implements a programming model in C++ for writing performance portable applications targeting all major HPC platforms. Homepage: https://github.com/kokkos/kokkos Tags: None Preferred version: 2.9.00 https://github.com/kokkos/kokkos/archive/2.9.00.tar.gz Safe versions: develop [git] https://github.com/kokkos/kokkos.git on branch develop 2.9.00 https://github.com/kokkos/kokkos/archive/2.9.00.tar.gz 2.8.00 https://github.com/kokkos/kokkos/archive/2.8.00.tar.gz ...
spack versions
Available versions are listed by the versions command
> spack versions kokkos ==> Safe versions (already checksummed): develop 2.9.00 2.8.00 2.7.24 2.7.00 2.5.00 2.04.11 2.04.04 2.04.00 2.03.13 2.03.05 2.03.00 2.02.15 2.02.07 ==> Remote versions (not yet checksummed): 3.1.01 3.1.00 3.0.00 2.6.00
spack find
The command find checks if a package is already installed
> spack find -l elpa ==> 9 installed packages -- linux-sles15-haswell / gcc@8.4.0 ----------------------------- o5ty6jb elpa@2017.11.001 -- linux-sles15-haswell / intel@19.0.5.281 ---------------------- vfxqn5c elpa@2018.11.001 zhoqg3p elpa@2019.05.002 bwykpas elpa@2019.11.001 nygb6zs elpa@2020.05.001 bhl4qmn elpa@2018.11.001 d2yg2xi elpa@2019.05.002 qztc5oz elpa@2019.11.001 i3gjkmf elpa@2020.05.001
shows the installed elpa versions and the compiler that has been used (gcc/intel). The -l option provides the unique hash for each package, which is also included in the name of the installation directory (check with > spack find -lp elpa
).
spack compilers
The compilers command lists the available compilers in spack
> spack compilers ==> Available compilers -- gcc sles15-x86_64 -------------------------------------------- gcc@9.3.0 gcc@8.4.0 gcc@7.5.0 -- intel sles15-x86_64 ------------------------------------------ intel@19.1.1.217 intel@19.0.5.281 intel@19.0.4.243 intel@18.0.4
These are the preconfigured ones. You may add your own using > spack compiler add /path/to/compiler
Installing Software with Spack
When installing software with spack in your home directory, you can make use of the resources and versions provided by LRZ.
spack spec
The spec command shows you the dependency tree of a given package
> spack spec -I libvdwxc Input spec -------------------------------- - libvdwxc Concretized -------------------------------- - libvdwxc@0.4.0%gcc@8.4.0+mpi~pfft arch=linux-sles15-haswell [^] ^fftw@3.3.8%gcc@8.4.0+mpi~openmp~pfft_patches precision=double,float arch=linux-sles15-haswell [^] ^intel-mpi@2019.7.217%gcc@8.4.0 arch=linux-sles15-haswell
The libvdwxc library depends on fftw und intel-mpi. The library itself is not yet installed (-) but both fftw and intel-mpi are already available via the upstream LRZ installation ([^]). A package that you have installed locally in your home directory with spack is marked by [+].
spack install
> spack install libvdwxc ==> intel-mpi is installed in an upstream Spack instance at /dss/dsshome1/lrz/sys/spack/release/20.1/opt/haswell/intel-mpi/2019.7.217-gcc-2vt6dak ==> fftw is installed in an upstream Spack instance at /dss/dsshome1/lrz/sys/spack/release/20.1/opt/haswell/fftw/3.3.8-gcc-vnifeuh ==> Installing libvdwxc ==> Searching for binary cache of libvdwxc ==> Finding buildcaches in /lrz/sys/spack/cache/build_cache ==> No binary for libvdwxc found: installing from source ==> Fetching file:///lrz/sys/spack/cache/libvdwxc/libvdwxc-0.4.0.tar.gz curl: (37) Couldn't open file /lrz/sys/spack/cache/libvdwxc/libvdwxc-0.4.0.tar.gz ==> Fetching from file:///lrz/sys/spack/cache/libvdwxc/libvdwxc-0.4.0.tar.gz failed. ==> Fetching file:///lrz/sys/spack/cache/_source-cache/archive/35/3524feb5bb2be86b4688f71653502146b181e66f3f75b8bdaf23dd1ae4a56b33.tar.gz curl: (37) Couldn't open file /lrz/sys/spack/cache/_source-cache/archive/35/3524feb5bb2be86b4688f71653502146b181e66f3f75b8bdaf23dd1ae4a56b33.tar.gz ==> Fetching from file:///lrz/sys/spack/cache/_source-cache/archive/35/3524feb5bb2be86b4688f71653502146b181e66f3f75b8bdaf23dd1ae4a56b33.tar.gz failed. ==> Fetching https://launchpad.net/libvdwxc/stable/0.4.0/+download/libvdwxc-0.4.0.tar.gz ############################################################################################################################# 100.0% ==> Staging archive: /gpfs/scratch/<project>/<uid>/<uid>/x86_avx2/spack-stage-libvdwxc-0.4.0-4eofava5vqw2kb35cdeocgq7fylmrfvk/libvdwxc-0.4.0.tar.gz ==> Created stage in /gpfs/scratch/<project>/<uid>/<uid>/x86_avx2/spack-stage-libvdwxc-0.4.0-4eofava5vqw2kb35cdeocgq7fylmrfvk ==> No patches needed for libvdwxc ==> Building libvdwxc [AutotoolsPackage] ==> Executing phase: 'autoreconf' ==> Executing phase: 'configure' ==> Executing phase: 'build' ==> Executing phase: 'install' ==> Successfully installed libvdwxc Fetch: 0.63s. Build: 24.30s. Total: 24.93s. [+] /path/to/home/spack/opt/linux-sles15-haswell/libvdwxc/0.4.0-gcc-8.4.0-4eofava
- line 2+3 : packages are taken from the LRZ installation
- line 8-13: spack checks if the source tar ball is available in the LRZ cache (not availabe)
- line 14: download from external site
- line 16-17: a build directory is created in
$TMPDIR
- line 18-24: library is built and installed
- line 26: installation location of the library in
$HOME/spack/opt/...
Providing source files
After a package has been downloaded, the soure files are stored in $HOME/spack/cache
Spack will search this cache next time you install the software and obtain the source from there:
> spack uninstall -ay libvdwxc ... > spack install libvdwxc ... ==> Using cached archive: /path/to/home/spack/cache/_source-cache/archive/35/3524feb5bb2be86b4688f71653502146b181e66f3f75b8bdaf23dd1ae4a56b33.tar.gz
Here, the source tar ball is stored indexed by its sh256 checksum. Sometimes you will have to supply the source tar ball by hand, either because it is commercial software or the download is not reachable due to firewall restrictions. You could follow the checksum caching scheme here, but more user friendly is to place the source in $HOME/spack/cache/<package name>/<package>-<version>.<suffix>
. You can deduce the correct naming scheme from the error message of an unsuccessful installation attempt.
Example:
> mkdir spack/cache/libvdwxc/ > mv /$HOME/spack/cache/_source-cache/archive/35/3524feb5bb2be86b4688f71653502146b181e66f3f75b8bdaf23dd1ae4a56b33.tar.gz spack/cache/libvdwxc/libvdwxc-0.4.0.tar.gz > spack uninstall -ay libvdwxc ==> Successfully uninstalled libvdwxc@0.4.0%gcc@8.4.0+mpi~pfft arch=linux-sles15-haswell/4eofava > spack install libvdwxc ... ==> Using cached archive: /path/to/home/spack/cache/libvdwxc/libvdwxc-0.4.0.tar.gz
Generating Modules for Your Spack Software
The module command triggers the generation of tcl environment modules.
> spack module tcl refresh libvdwxc ==> You are about to regenerate tcl module files for: -- linux-sles15-haswell / gcc@8.4.0 ----------------------------- 4eofava libvdwxc@0.4.0 ==> Do you want to proceed? [y/n] y ==> Regenerating tcl module files > module show libvdwxc ------------------------------------------------------------------- /path/to/home/spack/modules/x86_avx2/linux-sles15-haswell/libvdwxc/0.4.0-gcc8-impi: module-whatis {Portable C library of density functionals with van der Waals interactions for density functional theory} conflict libvdwxc prepend-path PATH /path/to/home/spack/opt/linux-sles15-haswell/libvdwxc/0.4.0-gcc-8.4.0-4eofava/bin prepend-path LD_LIBRARY_PATH /path/to/home/spack/opt/linux-sles15-haswell/libvdwxc/0.4.0-gcc-8.4.0-4eofava/lib prepend-path LIBRARY_PATH /path/to/home/spack/opt/linux-sles15-haswell/libvdwxc/0.4.0-gcc-8.4.0-4eofava/lib prepend-path PKG_CONFIG_PATH /path/to/home/spack/opt/linux-sles15-haswell/libvdwxc/0.4.0-gcc-8.4.0-4eofava/lib/pkgco> prepend-path CMAKE_PREFIX_PATH /path/to/home/spack/opt/linux-sles15-haswell/libvdwxc/0.4.0-gcc-8.4.0-4eofava/ setenv LIBVDWXC_BASE /path/to/home/spack/opt/linux-sles15-haswell/libvdwxc/0.4.0-gcc-8.4.0-4eofava ------------------------------------------------------------------
- line 1:
spack module tcl refresh
will regenerate all module file. We have configured spack such that modules will only be generated for explicitly installed packages. - line 11: modules are installed in
$HOME/spack/modules/$LRZ_INSTRSET/<architecture>/<package>/<version>
- line 15-20: the default module generated by spack sets the most important path and environment variables. If pgk-config support is available for a package the path is also included.
The path to the modules is added to $MODULEPATH when you load the user_spack module, but only if it already exists. So your new modules may only be visible the next time you load user_spack. You can check if the path is included with the command
> module use Search path for module files (in search order): /path/to/home/spack/modules/x86_avx2/linux-sles15-haswell /lrz/sys/spack/staging/20.1.1/modules/haswell/linux-sles15-haswell /lrz/sys/spack/staging/20.1.1/modules/x86_64/linux-sles15-x86_64
There, the path to spack/modules
should appear in the first line.
If you want to use these modules without loading spack_user first, you can add a
module use $HOME/spack/modules/$LRZ_INSTRSET/linux*
to your ~/.bashrc
file (~/.cshrc
, correspondingly).
Software already existing in the Software Stack
Sometimes you wish to create a module for software we have indeed installed, but don't provide a module for it (for whatsoever reason). Still, all is not lost. You can create a module also from those.
Ideally, you figured out with "spack find -pL <software name>
" whether this software (library, e.g.) is installed, where it is installed, and its hash. Please, check the correctness of OS version, hardware, compiler! For instance, on CoolMUC-2, let's check for udunits
:
> module load user_spack > spack find -pL udunits ==> 4 installed packages -- linux-sles15-haswell / gcc@11.2.0 ---------------------------- y2lpepnvxom7pqefxejnmgiqqzrbvhj3 udunits@2.2.28 /dss/dsshome1/lrz/sys/spack/release/22.2.1/opt/haswell/udunits/2.2.28-gcc-y2lpepn -- linux-sles15-haswell / intel@21.4.0 -------------------------- 42k5yhzgvhf526bk6zunvswtiq6h3jsb udunits@2.2.28 /dss/dsshome1/lrz/sys/spack/release/22.2.1/opt/haswell/udunits/2.2.28-intel-42k5yhz -- linux-sles15-skylake_avx512 / gcc@11.2.0 --------------------- g3vblls2sjgk7stxnbe4aqerh4dshonl udunits@2.2.28 /dss/dsshome1/lrz/sys/spack/release/22.2.1/opt/skylake_avx512/udunits/2.2.28-gcc-g3vblls -- linux-sles15-skylake_avx512 / intel@21.4.0 ------------------- 4ktifb7jmaqqll6kzxkfww3h3kbiudxz udunits@2.2.28 /dss/dsshome1/lrz/sys/spack/release/22.2.1/opt/skylake_avx512/udunits/2.2.28-intel-4ktifb7
We are on SLES 15, Haswell (CoolMUC-2), and look for gcc/11.2.0 compiled libraries (if it matters ... ). So, y2lpepnvxom7pqefxejnmgiqqzrbvhj3
is the desired hash (you can also use the first 7 characters or so, as long as it is unique).
Let's assume, we wish our modules be located in ~/modules
(it makes sense to differentiate also the spack version ... but we skip this for sake of illustration). So, we can create the module file as follows:
> mkdir -p ~/modules > spack -c"modules:tcl:blacklist_implicits:False" -c"modules:default:roots:tcl:$HOME/modules" module tcl refresh --upstream-modules /y2lpepnvxom7pqefxejnmgiqqzrbvhj3
Some explanations:
1) The -c options are necessary to overwrite the defaults. Some modules are explicitly blacklisted by us. No need to live with that for you! Furthermore, user_spack
makes a different directory the default module path for each user. That's for convenience ... if you don't want to think much about that. But if you do, you can also change that, as shown. (Please feel free to parse the spack documentation for more useful features!)
2) We vote for using the hashes, although in many cases, also the package names work ... maybe but for the price that you create modules also for OS versions/hardware/compilers you don't need.
Usage is then simple:
> module use -p ~/modules > module load udunits
We recomment also to consider module collections (see module help
!) for convenient restoring complete environments (including the insertion of module search paths). Especially for handling many modules, this can really accelerate your workflow!
Creating Your Own Package
Sometimes package.py files, which describe how to build a package, don't use the most recent version, are broken, or don't exist in the first place for your package. When you fix existing packages or create new packages you need to store them in your private repository.
spack repo
The first step ist to create your own repository
> mkdir ~/spack/repos > spack repo create ~/spack/repos/mine ==> Created repo with namespace 'mine'. ==> To register it with spack, run this command: spack repo add /path/to/home/spack/repos/mine
Now we have a repository with the name 'mine'. Next we need to register it, following the instructions in the last line
> spack repo add ~/spack/repos/mine ==> Added repo with namespace 'mine'. > spack repo list ==> 4 package repositories. mine /path/to/home/spack/repos/mine lrz /lrz/sys/spack/release/20.1/repos/lrz backport /lrz/sys/spack/release/20.1/repos/backport builtin /dss/dsshome1/lrz/sys/spack/user/release/20.1/spack/var/spack/repos/builtin
This package repo mine
was now added to the list of available repo from spack itself builtin
and from the LRZ provided repos lrz
and backport
.
spack edit / spack create
If you want to fix a package from the preconfigured repos, you copy to package to your own repo
> cp -r /path/to/repo/packages/libxc ~/spack/repos/mine/packages/ > spack edit libxc 1 # Copyright 2013-2020 Lawrence Livermore National Security, LLC and other 2 # Spack Project Developers. See the top-level COPYRIGHT file for details. 3 # 4 # SPDX-License-Identifier: (Apache-2.0 OR MIT) 5 6 from spack import * 7 8 9 class Libxc(AutotoolsPackage): 10 """Libxc is a library of exchange-correlation functionals for "~/spack/repos/mine/packages/libxc/package.py" 91L, 3618C
The spack edit
command opens the package file from the 'mine' repo in vim
or $EDITOR
.
If you create a new package with spack create, the package will be generated by default in the first repo, thus, in mine. For instructions and details or spack create please see the documentation https://spack-tutorial.readthedocs.io/en/latest/tutorial_packaging.html .
Configuring Your Spack Instance
LRZ provides a configuration that is very similar to the one the software stack was built with. You may want to change some or all of these settings to serve your needs, e.g. for the package selection, generation of modules, etc. Your individual configuration files are stored in the directory ~/.spack/. With the spac repo add command, you may already have created your own ~/.spack/repos.yaml file there. User config files take precedence over system provided config file, that is they are loaded after the system config files and overwrite their settings.
spack config
Configuration files can be viewed and modified with the config command. Taking up our repos expample
> spack config blame repos --- repos: /path/to/home/.spack/linux/repos.yaml:2 - /path/to/home/spack/repos/mine /dss/dsshome1/lrz/sys/spack/user/release/20.1/spack/etc/spack/linux/repos.yaml:2 - /lrz/sys/spack/release/20.1/repos/lrz /dss/dsshome1/lrz/sys/spack/user/release/20.1/spack/etc/spack/linux/repos.yaml:3 - /lrz/sys/spack/release/20.1/repos/backport /dss/dsshome1/lrz/sys/spack/user/release/20.1/spack/etc/spack/linux/repos.yaml:4 - $spack/var/spack/repos/builtin
we see that the 'mine' repo stems form the private config file in ~/.spack/linux/, whereas the other repos are configured in the spack site directory. All other config files reside in the same .../etc/spack/linux/
directory. You can either create new config files from scatch in ~/.spack/linux/ or copy them from the site directory and edit them. The command
> spack config --scope=site/linux edit --print-file modules /dss/dsshome1/lrz/sys/spack/user/release/20.1/spack/etc/spack/linux/modules.yaml
gives you the full path. The --scope=site/linux
option points to the site configuration (default are the user's config files).
You can edit your config files directly with spack. Check out
> spack config edit -h usage: spack config edit [-h] [--print-file] [SECTION] positional arguments: SECTION configuration section to edit. options: compilers, mirrors, repos, packages, modules, config, upstream
For the syntax and structure of these configuration files, please refer to the spack documentation https://spack-tutorial.readthedocs.io/en/latest/tutorial_configuration.html
Example: Generating modules for upstream packages
For packages that have been implicitly installed in the upstream spack stack no modules are generated by default. You can configure your `user_spack` such that you generate modules in your $HOME directory. Call
> spack config edit modules
and enter
modules: tcl: blacklist_implicits: False hash_length: 7
(ends up in `~/.spack/modules.yaml`). Now modules are generated for all installed modules (blacklisting of implitcits turned off) and each newly created module gets a hash suffix of length 7 to avoid naming conflicts. Modules are generated with
> spack module tcl refresh --upstream-modules
For a more fine grained control of the modules to be generated you could use the following scheme
modules: tcl: blacklist_implicits: False hash_length: 7 blacklist: [ '%intel', '%gcc' ] whitelist: - cmake
The blacklist blocks all packages installed with intel
and gcc
compilers (essentially all) and the whitelist unblocks all versions of cmake
.
Installing Software at a central place (e.g. a shared DSS container)
If you want to install a custom version of a spack package using user_spack
but you want also others to be able to use the installation, you can tweak the installation and module path for user_spack
. For this we assume that everyone already has access to a shared folder /path/to/your/shared/folder
(see also FAQ: Sharing files with other users ).
We create the necessary folders:
module load user_spack mkdir -p ~/.spack # just to be sure mkdir -p /path/to/your/shared/folder/user_spack/opt/ mkdir -p /path/to/your/shared/folder/user_spack/modules/
Then we edit the local spack config:
> spack config edit config
and add
config: install_tree: root: /path/to/your/shared/folder/user_spack/opt/
Similarly:
> spack config edit modules
and add
modules: default: enable: - tcl roots: tcl: /path/to/your/shared/folder/user_spack/modules/
Now we install our package (which is not already installed in the central installation). Here we use zlib
without shared libraries for demonstation:
> spack install zlib~shared ==> Installing zlib-1.2.13-hjfs3ik6vnhwotitnkyzjqyzafdv2hln ==> No binary for zlib-1.2.13-hjfs3ik6vnhwotitnkyzjqyzafdv2hln found: installing from source ==> Using cached archive: file:///lrz/sys/spack/cache/_source-cache/archive/b3/b3a24de97a8fdbc835b9833169501030b8977031bcb54b3b3ac13740f846ab30.tar.gz ==> No patches needed for zlib ==> zlib: Executing phase: 'edit' ==> zlib: Executing phase: 'build' ==> zlib: Executing phase: 'install' ==> zlib: Successfully installed zlib-1.2.13-hjfs3ik6vnhwotitnkyzjqyzafdv2hln Stage: 0.27s. Edit: 0.64s. Build: 0.64s. Install: 0.05s. Total: 1.65s [+] /path/to/your/shared/folder/user_spack/opt/linux-sles15-haswell/zlib/1.2.13-gcc-12.2.0-hjfs3ik
We generate the module(s):
> spack modules tcl refresh -y
and we should see the modules:
> module use -p /path/to/your/shared/folder/user_spack/modules/linux-sles15-haswell # "linux-sles15-haswell" could also be a bit different depending on the system your are on and the user_spack version. The correct path to use contains the folder called "zlib" > module av zlib ---------------------------------------------------- /path/to/your/shared/folder/user_spack/modules/linux-sles15-haswell ----------------------------------------------------- zlib/1.2.13-gcc12
As we assumed that your colleagues already have access to the shared folder, all they should have to do is
module use -p /path/to/your/shared/folder/user_spack/modules/linux-sles15-haswell # as explained above, the subfolder of the "modules" folder can differ for your setup
and then load the wanted module. If it does not work, please re-check the permissions of /path/to/your/shared/folder/user_spack/
and its subfolders and correct if necessary.