Default behaviour

All environment variables that are set in the shell at Job submission will be propagated to the sbatch script and will also be seen by commands run via srun . This includes for example:

  • all env vars defined in your ~/.bashrc , .bash_profile and ~/.profile that are loaded when you log in interactively via SSH.
  • all env vars set either manually or via modules
  • all env vars set by commands executed during or after login (e.g. anaconda/miniconda)

Avoid letting (ana|mini)conda write to your ~/.bashrc by invoking conda shell bash or similar. Always load via Lmod modules either before submitting Jobs or (preferred) in your sbatch script!

Unfortunately not every shell is the same, it can be a login or non-login shell, an interactive or non-interactive shell, and might be a remote shell or normal shell if SSH was involved in the execution or not. Depending on the type of shell, certain dotfiles (~/.bashrc , .bash_profile, ~/.profile ) are loaded or not.

For example: 

  • When you login one of the login nodes via SSH, a remote iteractive login  shell is invoked, .bash_profile  and ~/.profile will be loaded, but ~/.bashrc will not.
  • When you submit a Jobscript via sbatch, the Slurm Step Daemon invokes a normal non-login non-interactive shell and no dotfile will be loaded (env vars may be interited from the submitting shell).
  • When you invoke an interactive shell (e.g. via srun --time=1-0 --pty bash ), a normal non-login interactive shell is started, and just ~/.bashrc is loaded.

Fortunately, there is a ~/.profile dotfile by default, which also loads ~/.bashrc if it exists. So in all cases where any file will be loaded, ~/.bashrc will be loaded as well!

The recommended way to define aliases, environement variables and functions is to use ~/.bashrc exclusively.



Clean environments

Slurm offers several ways to start with no environments at all or clean environments (state right after login) with two Sbatch options: --export and --get-user-env .

  • --export={[ALL,]<environment_variables>|ALL|NIL|NONE}
    Identify which environment variables from the submission environment are propagated to the launched application. Note that SLURM_* variables are always propagated.
    • --export=ALL
      Default mode if --export is not specified. All of the user's environment will be loaded (either from the caller's environment or from a clean environment if --get-user-env is specified).
    • --export=NIL
      Only SLURM_* variables from the user environment will be defined. User must use absolute path to the binary to be executed that will define the environment. User can not specify explicit environment variables with "NIL". Unlike NONE, NIL will not automatically create a user's environment using the --get-user-env mechanism. 
    • --export=NONE
      Only SLURM_* variables from the user environment will be defined. User must use absolute path to the binary to be executed that will define the environment. User can not specify explicit environment variables with "NONE". However, Slurm will then implicitly attempt to load the user's environment on the node where the script is being executed, as if --get-user-env was specified.

      This option is particularly important for jobs that are submitted on one cluster and execute on a different cluster (e.g. with different paths). To avoid steps inheriting environment export settings (e.g. "NONE") from sbatch command, the environment variable SLURM_EXPORT_ENV should be set to "ALL" in the job script.
    • --export=[ALL,]<environment_variables>
      Exports all SLURM_* environment variables along with explicitly defined variables. Multiple environment variable names should be comma separated. Environment variable names may be specified to propagate the current value (e.g. "--export=EDITOR") or specific values may be exported (e.g. "--export=EDITOR=/bin/emacs"). If "ALL" is specified, then all user environment variables will be loaded and will take precedence over any explicitly given environment variables.

      Example: --export=EDITOR,ARG1=test

      In this example, the propagated environment will only contain the variable EDITOR from the user's environment, SLURM_* environment variables, and ARG1=test.

      Example: --export=ALL,EDITOR=/bin/emacs

      There are two possible outcomes for this example. If the caller has the EDITOR environment variable defined, then the job's environment will inherit the variable from the caller's environment. If the caller doesn't have an environment variable defined for EDITOR, then the job's environment will use the value given by --export. 


Environment best practises

Only use ~/.bashrc to modify your default environment!
Do not modify or delete ~/.profile !
Do not define aliases or functions that override builtin bash commands or existing commands. Check via type <command> if this name is already taken.

and most importantly

Keep your ~/.bashrc minimal!


part of icons are from social media marketing icons Social media marketing icons created by Freepik - Flaticon