Python on HPC systems
Preface
We provide several ways of using python at LRZ which have all different ways and ability for customization by the user
Python installed
OS installed python (likely wrong choice)
After login without loading modules or custom environments an OS-provided python will be available.
which python
In case the answer is /usr/bin/python
, then this is the version of python that comes by the OS. It is rather old but stable.
Typically this isn't the right choice for your scientific workload: probably not to use!
Module provided python
Check with the module avail command for the actually available version. Like,
module av python
---------------------------------------------- /lrz/sys/spack/release/22.2.1/modules/x86_64/linux-sles15-x86_64 ---------------------------------------------- python/3.7.11-base python/3.7.11-extended python/3.8.11-base(default) python/3.8.11-extended --------------------------------------------------------- /lrz/sys/share/modules/files_sles15/tools ---------------------------------------------------------- python/2.7_intel python/3.6_intel
Python compiled within the spack-stack (module-default)
Starting June 2021, two types of python modules are provided via the spack stack
python/<version>-base
python/<version>-extended
The version that is used by applications provided by spack is set to default.
The flavors (base & extended) differ on the libraries provided with them (check `ls $PYTHON_BASE/.spack/py-*
` after loading the module):
<version>-base
comes with only a few libraries (e.g.ipython
,autopep8
, ...)<version>-extended
additionally provides libraries likempi4py
,numpy
, andscipy
that have been compiled against Intel-MPI & Intel-MKL, respectively.
- these python installations are meant to be complemented with additional libraries via
pip
within your own user-space if required py your python- workflow or code
typically this is done inside a virtual-environment (see next section)
PIP: extending python in a virtual environment using pip
In case you need additional packages for your work - you may use PIP inside a virtual environment and install python-packages in addition inside such a virtual environment
# e.g.: module load python/3.8.11-base python3 -m venv ~/venv/py38 . ~/venv/py38/bin/activate pip3 install psutil
python3 # alternative python
Python 3.8.11 (default, Jan 26 2022, 17:51:41) [GCC 11.2.0] on linux Type "help", "copyright", "credits" or "license" for more information. >>>
then you can load the just installed psutil inside python
import psutil
You may find about which packages may be installed using pip-command in a shell or consult python-package-index on the web
you may deactivate your virtual-environment with
deactivate
Intel python modules (treated as deprecated from our side)
Two legacy modules with conda-based Intel-Python (currently versions 2.7_intel
and 3.6_intel
) are provided. They serve to support older python programs with a working version. Intel python includes support for Intel-MPI and Intel-MKL.
Note that python/2.7_intel
was the default module in the past. From June 2021 this is no longer the case and you have to specify this version explicitly upon loading. This is particularly important if your python program is not python 3.x compatible.
The version is provided 'as is'
Conda: Installing python & python packages using conda
With spack stack 22.2.1 modules for anaconda
and miniconda
are provided. After loading either of these you can download the latest python & python packages via conda. For that, you have to generate your own environment by
module load anaconda3 # alternative: module load miniconda3 conda create -n py38 python=3.8
now you can activate your environment ...
and install the package you like
source activate py38 conda install numpy
then you can load the just installed numpy inside python
python3 # alternative python
Python 3.8.10 | packaged by conda-forge | (default, May 11 2021, 07:01:05) [GCC 9.3.0] on linux Type "help", "copyright", "credits" or "license" for more information. >>>
import numpy
Find available packages via conda-command in your shell or web-search or conda-related documentation on the web (e.g. conda-forge-repo, conda-intel-repo)
deactivate conda-environment with
conda deactivate
Python on SuperMUC-NG
SuperMUC-NG does not allow for outgoing internet connections and therefore conda does not work out of the box (same for pip or R-CRAN) and you have to use a tunnel or use conda pack.
Please consult our FAQ section: FAQ: Python and R on SuperMUC-NG by using conda