Slurm tutorial

Submitting jobs (sbatch)

To use the compute nodes in our HPC clusters, you submit jobs to the batch job scheduling system Slurm. As a user you may want to become familiar with the following commands: sbatch, srun, scancel, squeue, sacct and ????, which are briefly described in the following sections.

For more information see the respective man pages of these commands. Detailed information may be found in the Slurm documentation pages.

The command sbatch is used to submit jobs to the batch-system using the following syntax:

sbatch [options] [job_script [ job_script_arguments ...]]

where job_script represents the path to (or if it's in the current working directory the name of) a simple shell script containing the commands to be run on the cluster nodes. If no file is specified, sbatch will read a script from standard input.

The first line of this script needs to start with #! followed by the path to an interpreter. For instance #!/bin/sh or #!/bin/bash (or any other available shell of your taste) but note that we currently only support Bash (/bin/bash). Your script may use common Bash functionality such as I/O redirection using the < and > characters, loops, case constructs etc., but please keep it simple. If your setup uses a different shell or needs a complex script, simply call your script from within the batch job script.

Slurm will start your job on any nodes that have the necessary resources available or put your job in a waiting queue until requested resources become available. Slurm uses recent consumption of resources in deciding which waiting jobs to start first ("fair share" scheduling). After submitting your job, you may continue to work or log out - job scheduling is completely independent of interactive work. The only way to stop a job after it has been submitted is to use the scancel command described below.

If you submit more than one job at the same time, you need to make sure that individual jobs (which may be executed simultaneously) do not interfere with each other, e.g. by writing to the same files.

The options tell the sbatch command how to behave: job name, use of main memory and run time, parallelization method, etc.

There are two ways of supplying these options to the sbatch command:

Method 1:

You may add the options directly to the sbatch command line, like:

sbatch -J job_name --ntasks=number_of_tasks --ntasks-per-node=number_of_tasks_per_node --cpus-per-task=number_of_cpus_per_task --mem-per-cpu=memory_per_cpu job_script [ argument ... ]

Method 2 (recommended):

Add the sbatch options to the beginning of your job_script, one option per line.

sbatch job_script [ argument ... ]

Note that the lines prefixed with #SBATCH are parsed by the sbatch command, but are treated as comments by the shell.

Taking above example, the contents of job_script would look like:

#!/bin/bash

#SBATCH -J job_name   #SBATCH --job-name=job_name is possible as well
#SBATCH --ntasks=number_of_tasks
#SBATCH --ntasks-per-node=number_of_tasks_per_node
#SBATCH --cpus-per-task=number_of_cpus_per_task
#SBATCH --mem-per-cpu=memory_per_cpu

./your_commands

If you give conflicting options both in the job file and the sbatch command line, the command line options take precedence. So you can use options in the job script to supply defaults that may be overriden in the sbatch command line.

Overview of commonly used options to sbatch

TODO stack=size[m|g] 

Job submission examples

Parallel MPI jobs

The contents of your job script may look like this:
(if you just copy&paste this example please be aware of line breaks and special characters)

#!/bin/bash

# Name of your job.
#SBATCH -J name

# Send status information to this email address. 
#SBATCH --mail-user=Karl.Mustermann@xxx.com

# Send an e-mail when the job has finished or failed. 
#SBATCH --mail-type=END,FAIL

# In this example we are allocating ressources for 100 MPI processes/tasks,
# placing exactly 10 tasks on each of 10 separate nodes like this:
#SBATCH --ntasks-per-node=10
#SBATCH --nodes=10
#SBATCH --cpus-per-task=1

# This is the same as above but leaving the decision about the
# number of tasks on how many nodes to Slurm (aka "fillup"):
## #SBATCH --ntasks=100
## #SBATCH --cpus-per-task=1

# Specify the amount of memory given to each MPI process
# in the job.
#SBATCH --mem_per_cpu=1G

module purge
module load openmpi/xx.yy.zz

mpirun -np $SLURM_NTASKS ./your_mpi_executable [extra arguments]

Parallel OpenMP jobs

The contents of your job script may look like this:
(if you just copy&paste this example please be aware of line breaks and special characters)

#!/bin/bash

# Name of your job.
#SBATCH -J name

# Send status information to this email address. 
#SBATCH --mail-user=Karl.Mustermann@xxx.com

# Send an e-mail when the job has finished or failed. 
#SBATCH --mail-type=END,FAIL

# Allocate one task on one node and six cpus for this task
#SBATCH --ntasks=1
#SBATCH --ntasks-per-node=1
#SBATCH --cpus-per-task=6

# Allocate 12 Gigabytes for the whole node/task
#SBATCH --mem=12G

# tell OpenMP how many threads to start
export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK
./your_openmp_executable

Important: We have configured Slurm to use control groups in order to limit access of your job to memory and cpus.
If your job uses shared memory parallelization other than OpenMP, you should check that the number of CPU-intensive threads is consistent with the number of slots assigned to the job.
If you start more threads than you requested in the --cpus-per-task directive, these will be forced to run on the requested amount of cores so they will interfere with each other, possibly degrading the overall efficiency of your job.

Submitting interactive jobs (srun --pty)

The submission of interactive jobs is useful in situations where a job requires some sort of direct intervention.

This is usually the case for X-Windows applications or in situations in which further processing depends on your interpretation of immediate results. A typical example for both of these cases is a graphical debugging session.

Note: Interactive sessions are also particularly helpful for getting acquainted with the system or when building and testing new programs.

Interactive sessions might be sequential or parallel ones:

In a multi-node parallel interactive session you can use srun (or after loading an MPI module mpirun) to run programs on all nodes.

For using an X-Windows application you have to supply --x11 as a further option, e.g. like this:

srun --pty --x11 xterm -ls

Prepare your session as needed, e.g. by loading all necessary modules within the provided xterm and then start your program on the executing node.

Note: Make sure to end your interactive sessions (logging out or closing the xterm windows) as soon as they are no longer needed!

Monitoring jobs (squeue and sacct)

To get information about running or waiting jobs use

squeue [options] or sacct -X [options]

TODO:
Ausführlicher!
Optionen für Anzeige der Priorität, des voraussichtlichen Startdatums usw.

Deleting a job (scancel)

To delete a job which has the job identifier job-id, execute

scancel job-id