Using SLURM

The first step to taking advantage of our clusters using SLURM is understanding how to submit jobs to the cluster using SLURM.

Job submission scripts are nothing more than shell scripts that can have some additional "comment" lines added that specify option for SLURM. For example, this simple BASH script can be a job submission script:

#!/bin/bash

#SBATCH --output=slurm-%j.out
#SBATCH --nodes=2
#SBATCH --time=10:00
srun hostname
srun sleep 30

The "sleep 30" line is there just to keep this short scripts running a little longer for demonstration purposes. This script can now be submitted to SLURM using the SBATCH command:
$sbatch bash_hostname.sh
Submitted batch job 306
In the above output, the "306" is the job ID. Once the job is submitted, you can check the status of it using the squeue command:
squeue
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
306 all submit_t lcolbert R 0:07 2 node[01-02]
The "R" in the ST column means that the job is running. You may also see Pending "PD" which means that the job is awaiting resource allocation or Configuring "CF" which means that resources have been allocated but are waiting for them to become ready for use. By default, SLURM runs the job from the directory it was submitted from and writes both standard output and standard error into a file here. The output filename format will default to slurm-<jobid>.out. You can specify the filename for the output file, or separate output and error, with the flags --output=<filename pattern> and --error=<filename pattern>.
#!/bin/bash
#SBATCH --output=hello-%j.out
#SBATCH --error=hello-%j.err
#SBATCH --nodes=2
#SBATCH --time=10:00

srun hostname
srun sleep 30
You can add any switch to sbatch to your submission script this way. For example, here is a more complicated submission script to run a parallel job:
#!/bin/bash
#SBATCH --nodes=16
#SBATCH --ntasks-per-node=28
#SBATCH --time=1:00:00
#SBATCH --mail-type=begin
#SBATCH --mail-type=end
#SBATCH --mail-user=<user>@ias.edu
#SBATCH --export=all

module load openmpi/intel
srun ./xhpl

Specifying a wallclock time
When a runtime is not specified a default of 12 hours is assumed. You can specify requested time with "#SBATCH --time=<dd:hh:mm> or #SBATCH -t <dd:hh:mm>.

Requesting exclusive use of a node
There are some use cases where you will want to allocate jobs to nodes with no other users or processes running. Examples would be memory intensive jobs utilizing all of a nodes RAM or threaded programs not explicitly controlled by SLURM. In these cases you can request that your job has exclusive use of a node and that no other jobs (outside of your own) will be running on it at the same time.
#SBATCH --exclusive
#SBATCH --exclusive=user # restrict to user jobs

SLURM Tips
See only the jobs of a specific user
squeue -u <user>

Show estimated start time of a job
squeue --start

Show jobs running on specific nodes
squeue --nodelist=node01,node10