Slurm tutorial

To use the compute nodes in our Slurm clusters (HPC Cluster LEO5 and teaching cluster LCC2), you submit jobs to the batch job scheduling system Slurm (unlike LEO3e and LEO4, which still use SGE).

As a user you may want to become familiar with the following commands: sbatch, srun, scancel, squeue, sacct and scontrol, which are briefly described in the following sections.

For more information see the respective man pages of these commands. Detailed information may be found in the Slurm documentation pages.

For those of you who want to re-use their work we have installed a tool to convert SGE-job-scripts to their Slurm equivalent.
You call it like this:

/usr/site/hpc/bin/sge2slurm.py sge-jobscript.sge > slurm-jobscript.slurm

Important differences between Slurm and SGE

Essentially SGE and Slurm are very similar as both are batch schedulers. But apart from the different commands and their options (described below) there are these important design differences between our SGE-setup and our Slurm configuration:

• Memory: RSS vs. VSZ the memory-related options of Slurm apply to the Resident Set Size - RSS of your job, this is the amount of "real" memory that your programs are using.
  In SGE, the only way to specify your job's memory limit was its Virtual Memory Size - VSZ (option -l h_vmem). Many programs allocate huge amounts of virtual memory without ever using it, forcing you to overbook memory with SGE. As a result, when moving to Slurm, you may be able to considerably reduce your memory requirements down to the amount of actually needed RSS.
• Memory limit for multiple tasks: Similar to SGE, if you specify --mem-per-cpu for MPI jobs, each group of tasks running on the same host will run in a shared memory allocation whose size is the sum of the individual requested sizes. If tasks have varying memory needs and some of them exceed their limit, your job may or may not fail depending on how tasks are scattered across nodes.
• Resource-over-consumption: we have configured Slurm in a way that when your program uses more resources (CPUs and memory) than you requested in your job options it will behave in a way different from SGE:
  • If (only) a program's virtual memory size (VSZ) exceeds the memory limit, it will simply keep running with no adverse effects.
  • If a program's RSS exceeds the memory limit, it will not be terminated. We had to temporarily change this as a workaround due to a bug in the control group subsystem - see Known Problems! Instead, parts of its memory will be paged out (and in) to disk. We allow such paging events so an occasional overuse of memory will not teminate your job. However, if this happens often, it will substantially slow down your job, so you should definitely avoid this.
  • Slurm uses so called control groups to confine your programs to the requested number of CPUs (per node). I.e. if one of your program spawns more processes or threads then your requested number of CPUs it will not impair the performance of other jobs on the same node, but your job may run more slowly than expected.
• SGE's concept of job slots does not exist in Slurm. With Slurm, resource requests (such as CPUs, cores, etc.) always refer to physical resources. In particular, since we have permitted use of hardware-hyperthreading, each hardware thread counts as a CPU (consistent with /proc/cpuinfo). So on the Leo5 Slurm configuration, there are 2 CPUs per core, 32 cores per socket, 2 sockets per node, making 64 cores per node or 128 CPUs per node.
  Only certain workloads can profit from using multiple CPUs (i.e. hardware threads) per core. We encourage you to experiment to find out if your jobs run faster with hyperthreads enabled (use --hint=multithread to do that). By default only 1 CPU per core is used. To prevent other jobs from competing with the shared resources of this core, the entire core (i.e. both CPUs) will still be silently allocated to your job. As a result, twice as many CPUs as requested will be reported in non-hyperthreaded jobs.
• When you don't specify a run time on LEO5, it is set to 24 hours. You may specify a maximum of 10 days if needed. Please note that excessive run times negatively affect overall responsiveness of the cluster.
• In their long form, Slurm's command line options require the double hyphen and an equal-sign (e.g. -n 24 in long form is --ntasks=24). For mpirun, the double hyphen is optional.
• By default, all currently set environment variables (including loaded modules), are propagated to the job by the sbatch command. So we recommend using a module purge line in your job-scripts before loading the modules you need.
• Slurm jobs by default ignore your $HOME/.bashrc shell initialization script.
• The default working directory of your Slurm job is the current working directory of your sbatch command. SGE's -cwd option is no longer necessary.

Submitting jobs (sbatch)

The command sbatch is used to submit jobs to the batch-system using the following syntax:

sbatch [options] [job_script.slurm [ job_script_arguments ...]]

where job_script.slurm represents the (relative or absolute) path to a simple shell script containing the commands to be run on the cluster nodes. We recommend to use the suffix .slurm to distinguish from scripts intended for other uses. If no file is specified, sbatch will read a script from standard input.

The first line of this script needs to start with #! followed by the path to an interpreter. For instance #!/bin/sh or #!/bin/bash (or any other available shell of your taste) but note that we currently only support Bash (/bin/bash). Your script may use common Bash functionality such as I/O redirection using the < and > characters, loops, case constructs etc., but please keep it simple. If your setup uses a different shell or needs a complex script, simply call your script from within the batch job script.

Slurm will start your job on any nodes that have the necessary resources available or put your job in a waiting queue until requested resources become available. Slurm uses recent resource consumption in deciding which waiting jobs to start first (fair share scheduling). After submitting your job, you may continue to work or log out - job scheduling is completely independent of interactive work. The only way to stop a job after it has been submitted is to use the scancel command described below.

If you submit more than one job at the same time, you need to make sure that individual jobs (which may be executed simultaneously) do not interfere with each other by e.g. writing to the same files.

The options tell the sbatch command how to behave: job name, use of main memory and run time, parallelization method, etc.

There are two ways of supplying these options to the sbatch command:

Method 1:

You may add the options directly to the sbatch command line, like:

sbatch --job-name=job_name --ntasks=number_of_tasks --cpus-per-task=number_of_cpus_per_task --mem-per-cpu=memory_per_cpu job_script.slurm [ argument ... ]

Method 2 (recommended):

Add the sbatch options to the beginning of your job_script, one option per line.

Note that the lines prefixed with #SBATCH are parsed by the sbatch command, but are treated as comments by the shell.

Taking above example, the contents of job_script.slurm would look like:

#!/bin/bash

#SBATCH --job-name=job_name
#SBATCH --ntasks=number_of_tasks
#SBATCH --cpus-per-task=number_of_cpus_per_task
#SBATCH --mem-per-cpu=memory_per_cpu

./your_commands

If you give conflicting options both in the job file and the sbatch command line, the command line options take precedence. So you can use options in the job script to supply defaults that may be overriden in the sbatch command line.

Overview of commonly used options to sbatch

For details, please look at the sbatch documentation.

--job-name=name

--output=filename_pattern

--error=filename_pattern

--input=filename pattern

--chdir=directory

--export=NONE

--mail-user=email_address

--mail-type=[TYPE|ALL|NONE]

--time=time

--mem=size[K|M|G|T]

--mem-per-cpu=size[K|M|G|T]

--mem-per-gpu=size[K|M|G|T]

--ntasks=ntasks

--nodes=n[-m]
--ntasks-per-node=ntasks

--cpus-per-task=ncpus

--hint=multithread

--gpus=[type:]number

--array=m-n[:step][%maxrunning]

--test-only

A note on maximum stack size

Our systems are configured in a way that the maximum allowed size of the stack of your programs is unlimited (unlike the default in most Linux systems where it is limited to 8 MB). Most programs will not need this but some will benefit from it.
There are edge-cases where (FORTRAN?) programs will not work with an unlimited stack size. In that case please limit stack size in your job-script before calling that program. With e.g.

ulimit -s 80000

you will set the limit to about 80 MB (80000 kB). This works because as a user you are allowed to lower the limit anytime.

Running Parallel Jobs

The setup of parallel jobs submitted by sbatch --ntasks=n_tasks consists of two phases:

1. Starting the job script on the master node selected by the batch scheduler. This is the actual job.
2. Deploying the individual n_tasks tasks across the worker nodes selected by the batch system. In Slurm terms, this phase is called a job step. There may be multiple job steps in a job.

For starting a job step, you can...

• ... either use Slurm's srun command - this is now the preferred way -
• ... or use the mpirun command if supplied by the MPI implementation that you are using.

In our OpenMPI installations, we enable the legacy launchers option, so the mpirun command will be available, and it will correctly honor your resource requests given in your sbatch command or job script. Behind the scenes, mpirun will call srun. With other MPI implementations, in particular when they have not been linked against our Slurm-integrated OpenMPI, mpirun may be available but can have bizarre effects, including the following:

• all tasks run on the same CPU, causing a massive slow-down
• all tasks have rank 0
• no remote tasks can be started

For this reason, you should write new jobs using the srun command. The srun command largely accepts the same options as the sbatch command, but normally, it will take its resource information from environment variables set by sbatch.

Advantages of using the srun command:

• Full integration into the Slurm run time environment
• No need to load the correct MPI module if your binaries have been linked using the RPATH attribute
• Works for some MPI implementations that have not been integrated with Slurm, e.g. Anaconda's MPICH.

When you use the srun command, you may want to set some environment variables or use the following options in your batch script:

srun --export=ALL [...]

export SRUN_CPUS_PER_TASK=${SLURM_CPUS_PER_TASK:-1}

srun --cpus-per-task=${SLURM_CPUS_PER_TASK:-1} [...]

srun --mpi=pmi2 [...]

Job submission examples

Parallel MPI job

The contents of your job script may look like this:
(if you just copy&paste this example please be aware of line breaks and special characters)

#!/bin/bash

# Name of your job.
#SBATCH --job-name=name

# Send status information to this email address. 
#SBATCH --mail-user=Karl.Mustermann@xxx.com

# Send an e-mail when the job has finished or failed. 
#SBATCH --mail-type=END,FAIL

# Start an MPI job with 80 single threaded tasks
#SBATCH --ntasks=80
#SBATCH --cpus-per-task=1

# In this example we allocate ressources for 80 MPI processes/tasks,
# placing exactly 10 tasks on each of 8 separate nodes like this:
## #SBATCH --ntasks-per-node=10
## #SBATCH --nodes=8
## #SBATCH --cpus-per-task=1
# do this only when you have good reason to explicitly control
# task placement

# Specify the amount of memory given to each MPI process
# in the job.
#SBATCH --mem-per-cpu=1G

module purge
module load openmpi/xx.yy.zz

mpirun ./your_mpi_executable [extra arguments]

Parallel OpenMP jobs

The contents of your job script may look like this:
(if you just copy&paste this example please be aware of line breaks and special characters)

#!/bin/bash

# Name of your job.
#SBATCH --job-name=name

# Send status information to this email address. 
#SBATCH --mail-user=Karla.Musterfrau@xxx.com

# Send an e-mail when the job has finished or failed. 
#SBATCH --mail-type=END,FAIL

# Allocate one task on one node and six cpus for this task
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=6

# Allocate 12 Gigabytes for the whole node/task
#SBATCH --mem=12G

# it is no longer necessary to tell OpenMP how many software threads to start
####    export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK
./your_openmp_executable

Important: We have configured Slurm to use control groups in order to limit access of your job to memory and cpus.
If your job uses shared memory parallelization other than OpenMP, you should check that the number of CPU-intensive software threads is consistent with the number of slots assigned to the job.
If you start more software threads than you requested in the --cpus-per-task directive, these will be restricted to run on the requested amount of CPUs so they will interfere with each other, possibly degrading the overall efficiency of your job.

Hybrid MPI + OpenMP Parallel Job

#!/bin/bash

# Name of your job.
#SBATCH --job-name=name

# Send status information to this email address. 
#SBATCH --mail-user=Karl.Mustermann@xxx.com

# Send an e-mail when the job has finished or failed. 
#SBATCH --mail-type=END,FAIL

# Start an MPI job with 20 tasks with 4 software threads each
#SBATCH --ntasks=20
#SBATCH --cpus-per-task=4

# In this example we allocate ressources for 20 hybrid MPI+OpenMP tasks,
# placing exactly 4 tasks on each of 5 separate nodes like this:
## #SBATCH --ntasks-per-node=4
## #SBATCH --nodes=5
## #SBATCH --cpus-per-task=4
# do this only when you have good reason to explicitly control
# task placement

# Specify the amount of memory given to each MPI process
# in the job.
#SBATCH --mem-per-cpu=1G

module purge
# module load [....]

# let Slurm take care of both levels of parallelism
export SRUN_CPUS_PER_TASK=${SLURM_CPUS_PER_TASK:-1}
srun --export=ALL --mpi=pmi2 ./your_mpi_executable [extra arguments]

Heterogeneous Jobs

For some MPI applications, resource requirements of individual tasks may differ from each other. A typical use case is a main - worker setup, where the main task, running in MPI rank 0, coordinates the worker tasks, which are running in ranks 1 to n, totaling n+1 tasks.

Slurm supports this type of setup, but as of April 2023, the original documentation is incomplete and misleading. Use the information given below in addition to Slurm's documentation.

To support heterogeneous jobs, Slurm allows you to specify multiple groups of resource requirements, to which individual tasks are assigned in ascending rank order. Resource allocation groups are separated by a colon (:) in the command line and by the directive #SBATCH hetjob in the batch script.

The following template assumes a memory intensive main task and 20 CPU intensive worker tasks. Please note that contrary to above quoted documentation, the resource requirements passed to the srun command by default do not reflect the requirements specified in your batch script or sbatch command and thus must be repeated explicitly in the srun command. Please adjust the following template according to your own needs:

#!/bin/bash
#SBATCH some non-resource options as needed
#SBATCH --ntasks=1 --cpus-per-task=1 --mem-per-cpu=1200
#SBATCH hetjob
#SBATCH --ntasks=20 --cpus-per-task=4 --mem-per-cpu=50 --hint=multithread
#SBATCH more options as needed

# the following must be consistent with above requirements
resources=' --ntasks=1 --cpus-per-task=1 --mem-per-cpu=1200 : --ntasks=20 --cpus-per-task=4 --mem-per-cpu=200 --hint=multithread'

srun [your options] $resources your-mpi-program [ your arguments ... ]

This will run MPI rank 0 on a single CPU with 1200 MB RSS available. Ranks 1-20 will run on 4 CPUs each (hybrid MPI+OpenMP) and make use of hardware hyperthreads, effectively using both hyperthreads of each of the two cores assigned to each task. Memory available per task will be (50 MB times 4 CPUs) = 200 MB each.

Interactive jobs (srun --pty)

The submission of interactive jobs is useful in situations where a job requires some sort of direct intervention.

This is usually the case for X-Windows applications or in situations in which further processing depends on your interpretation of immediate results. A typical example for both of these cases is a graphical debugging session.

Note: Interactive sessions are also particularly helpful for getting acquainted with the system or when building and testing new programs.

Interactive sessions might be sequential or parallel:

In a multi-node parallel (aka distributed memory) interactive session you can use srun (or after loading an MPI module mpiexec) to run programs on all nodes.

For using an X-Windows application, supply --x11 as a further option, e.g. like this:

srun --pty --x11 xterm -ls

Prepare your session as needed, e.g. by loading all necessary modules within the provided xterm and then start your program on the executing node.

Note: Make sure to end your interactive session (logging out or closing the xterm window) as soon as it is no longer needed!

Monitoring jobs (squeue, scontrol, srun and sacct)

To get information about running or waiting jobs use

squeue [options]
or
sq [options]

The command squeue displays a list of all running or waiting jobs of all users.

The locally implemented command sq displays more fields than squeue does by default.

The (in our opinion) most interesting additional field is START_TIME which for pending jobs shows the date and time when Slurm plans to run this job. It is always possible that a job will start earlier but not (much) later.

Slurm calculates this field only once per minute so it might not contain a meaningful value right after submitting a job.

squeue and sq display the jobs of all users which might not always be what you want. So we created another shortcut for you:
squ
which is a shorter way of typing sq -u $USER and thus lists only the jobs belonging to you.

Get more detailed information about a particular pending or running job by
scontrol show job jobid.

You can further inspect a running job by "connecting" to it with this command:

srun --jobid=jobid --overlap --pty bash

This will open an interactive shell as a job step under an already allocated job. I.e. you will be able to see how your job is "behaving". For distributed memory jobs you will get a shell at the first node used by your job.

To get information about past jobs use

sacct -X [options]

Altering jobs (scontrol update)

You can change the configuration of pending jobs with

scontrol update job jobid SETTING=VALUE [...]

To find out which settings are available we recommend to first run
scontrol show job jobid.

If then for example you want to change the run-time limit of your job to let's say three hours you would use
scontrol update job jobid TimeLimit=3:00

Some adaption might require you to change more than one setting. If e.g. your Job is flexible wrt to the number of used tasks and nodes and you want to change those after having submitted a job you would have to run
scontrol update job jobid NumTasks=xx NumCPUs=xx NumNodes=y

Deleting jobs (scancel)

To delete pending or running jobs you have to look up their numerical job identifiers aka job-ids. You can e.g. use squeue or squ and take the value(s) from the JOBID column.
Then execute

scancel job-id [...]

and the corresponding job will be removed from the wait queue or stopped if it's running.
You can specify more than one job identifier after scancel.
If you want to cancel all your pending and running jobs without being asked for confirmation you may use
squ -h -o %i | xargs scancel.
Those options tell squeue to only output the JOBID column (-o %i) and to omit the column header (-h).