General
The NVIDIA CUDA Toolkit provides a development environment for creating high-performance, GPU-accelerated applications. The toolkit includes GPU-accelerated libraries, debugging and optimization tools, a C/C++ compiler, and a runtime library.
The main binary is nvcc , which is called a CUDA compiler driver. nvcc accepts both C/C++ as well as CUDA code but needs a host compiler (typically gcc or g++ from the GCC suite) for most applications. Further details on the nvcc compilation process can be found here.
Lmod modules
ml av cuda
At the time of writing there are the following cuda versions (version X.Y.Z means: major version X , minor version Y and bugfix version Z ) available:
| Lmod module name |
|---|
| cuda/10.2 |
| cuda/11.6.2 |
| cuda/11.8.0 |
| cuda/12.1.1 |
| cuda/12.2.2 (default) |
cuda/12.3.2 (consider loading cuda-compat/12.3.2 as well) |
cuda/12.4.1 (consider loading cuda-compat/12.4.1 as well) |
| cuda/12.5.1 (consider loading cuda-compat/12.5.1 as well) |
| cuda/12.6.1 (consider loading cuda-compat/12.6.1 as well) |
Do not specify the bugfix version number when loading the CUDA Toolkit
Usually, bugfix releases are installed close after release and replace the old releases. We therefore highly recommend to specify up to the minor version:
ml load cuda/12.2
CUDA Toolkit interoperability
Minor Version Compatibility
While CUDA Toolkit from the 10.x series always required upgrading both, the Toolkit and the GPU driver, Nvidia introduced Minor Version Compatibility with it's 11.x and 12.x series. That means that, applications compiled with any CUDA Toolkit release from within a CUDA major release series will run using any other CUDA version of the same major version (sometimes with limited feature-set, see caveat below) on systems having at least the minimum required driver version.
Caveat: Sometimes features introduced in a CUDA Toolkit version may actually span both the toolkit and the driver. In such cases an application that relies on features introduced in a newer version of the toolkit and driver may for example return the following error on older drivers: cudaErrorCallRequiresNewerDriver.
Forward Compatibility
The currently installed cuda driver is v535.129.03 (which comes with CUDA Toolkit 12.2.2, hence the default version). With every new CUDA Toolkit there is a new driver version available. However, on HPC clusters, the NVIDIA Drivers cannot be upgraded everytime a new Toolkit is release (typically every other month). This is why Nvidia has introduced Forward Compatibility to allow to only update the usermode part of the NVIDIA GPU driver. It’s mainly intended to support applications built on newer CUDA Toolkits to run on systems installed with an older NVIDIA Linux GPU driver from different major release families.
It may not be necessary to use the cuda-compat modules, however it is recommended to load it when using cuda toolkit versions newer than 12.2.2 for guaranteed compatibility.
Applications that compile device code to PTX will not work on older drivers and therefore require loading of the respective cuda-compat module.
Forward compatible usermode librarys are available using the cuda-compat Lmod modules:
| Lmod module name |
|---|
cuda-compat/12.3.2 |
cuda-compat/12.4.1 |
| cuda-compat/12.5.1 |
| cuda-compat/12.6.1 |
Binary Backward Compatibility
The CUDA driver maintains backward compatibility to continue support of applications built on older toolkits, i.e. applications built with older CUDA toolkits will continue to run even with newer toolkits. For more details, have a look at the respective version of the Online Documentation.
Host Compiler compatibility
nvcc of a specific CUDA Toolkit version is compatible with the following GCC and clang versions:
| CUDA Toolkit | GCC | clang | icc (not recommended) |
|---|---|---|---|
| 12.6 series | 6.x - 13.2 | 7.x - 18.0 | 2021.7 (intel/2022) |
| 12.5 series | 6.x - 13.2 | 7.x - 17.0 | 2021.7 (intel/2022) |
| 12.4 series | 6.x - 13.2 | 7.x - 17.0 | 2021.7 (intel/2022) |
| 12.3 series | 6.x - 12.2 | <=16 | 2021.7 (intel/2022) |
| 12.2 series | 6.x - 11.2 | <=15 | 2021.7 (intel/2022) |
12.1 series | 6.x - 11.2 | <=15 | 2021.7 (intel/2022) |
11.8 series | 6.x - 11.2 | <=14 | 2021.4 (intel/2021) |
11.6 series | 6.x - 11.2 | <=12 | 2021.4 (intel/2021) |
10.2 series | 4.4.7 - 8.x | <=8 | - |
Compatibility with HPC-SDK
The HPC-SDK brings its its own CUDA Toolkit versions incorporated, therefore loading nvhpc and cuda modules simultaneously is neither necessary nor allowed.
Compatibility with Intel compilers
While some Intel compiler versions are supported, the range is very narrow and has not been tested to work on LiCCA. Use with caution, but better use GCC compilers instead.
With Intel shifting away from their classic compilers to modern LLVM (clang) based compilers, the clang column applies for modern intel compilers (icx,icpx,ifx). In order to determine the underlying clang version, use icx -x c /dev/null -dM -E|grep clang
