FAQ: Using Dask and ipcluster on Linux Cluster
In order to use the load balancing features of dask (https://dask.org/) on the Linux Cluster, here is a recipe how to set it up:
- Using the conda command install:
$ module load anaconda3
$ conda create -n py38 python=3.8
$ conda install ipython
$ conda install ipcluster - now you are ready to run a ipcluster program on a single node on the Linux Cluster
Use this slurm batch file:
#!/bin/bash #SBATCH -J ipcluster #SBATCH --get-user-env #SBATCH --clusters=cm2_tiny #SBATCH --partition=cm2_tiny #SBATCH --nodes=1-1 #SBATCH --cpus-per-task=28 #SBATCH --time=08:00:00 module load slurm_setup source ~/miniconda39/bin/activate ipcluster start --daemonize echo "ipcluster started" sleep 10 echo "starting calculation" python -c "from ipyparallel import Client; c=Client(); print(c.ids)" ipcluster stop