Computing on the Slurm preview cluster

Important

This page describes the Slurm preview cluster. Users access this environment by logging in to login.hoffman2.oarc.ucla.edu. It runs a different operating system environment from the production cluster and uses Slurm instead of Altair Grid Engine. These instructions are for the Slurm preview cluster only. For the production Hoffman2 Cluster environment, use Computing with Altair Grid Engine on the production cluster, which uses Altair Grid Engine commands such as qrsh, qsub, qstat and qdel.

The Slurm preview cluster is separate from the production Altair Grid Engine scheduler and operating system environment. Use the Slurm commands on this page only after connecting to login.hoffman2.oarc.ucla.edu. Jobs submitted with Slurm are managed with Slurm job IDs and Slurm resource options; production Altair Grid Engine job IDs and command options do not apply.

This page describes:

Connecting to the Slurm preview cluster

Connect to the Slurm preview cluster with:

$ ssh login.hoffman2.oarc.ucla.edu

This login host is the clearest delimiter between the production and Slurm preview clusters:

Production and Slurm preview cluster differences

Area

Production cluster

Slurm preview cluster

Login environment

Production Hoffman2 login environment

login.hoffman2.oarc.ucla.edu

Scheduler

Altair Grid Engine

Slurm

Scheduler commands

qrsh, qsub, qstat, qdel

srun, salloc, sbatch, squeue, scancel

Operating system

Production operating system environment

New Slurm preview operating system environment

Shared user storage

$HOME, $SCRATCH and project directories, if any

Same $HOME, $SCRATCH and project directories, if any

Application location

Production application stack

New application stack under /u/local/apps

Note

Your files in $HOME, $SCRATCH and project directories, if any, are shared between the production and Slurm preview clusters. The operating system and application stack are different, so software that works in one environment may need to be rebuilt, reloaded or reconfigured in the other.

Slurm preview command equivalents

Use the scheduler command that matches the environment where you are working:

Production Altair Grid Engine and Slurm preview commands

Task

Production Altair Grid Engine

Slurm preview

Start an interactive session

qrsh

srun --pty bash -l or salloc

Submit a batch job

qsub

sbatch

Check jobs

qstat or myjobs

squeue or squeue --me

Cancel a job

qdel

scancel

Job script scheduler directives

#$

#SBATCH

Requesting Slurm preview resources

Slurm resource requests use Slurm options rather than Altair Grid Engine complexes. The most common requests are:

Common Slurm preview resource options

Resource

Slurm option

Example

Runtime

--time

--time=02:00:00

Memory

--mem or --mem-per-cpu

--mem=4G

CPU cores

--cpus-per-task, --ntasks or both

--cpus-per-task=4

GPU cards

--gres

--gres=gpu:1

Partition

--partition or -p

Use only when a Slurm preview partition has been provided for your work

Job name

--job-name

--job-name=myjob

Note

The production Altair Grid Engine options -l and -pe are not Slurm options. In the Slurm preview cluster, use Slurm resource flags such as --time, --mem, --cpus-per-task, --ntasks and --gres.

Requesting Slurm preview interactive sessions

For interactive work on the Slurm preview cluster, request resources with srun or allocate them first with salloc.

For example, to start an interactive shell with one CPU core, 4 GB of memory and a two-hour runtime:

$ srun --time=02:00:00 --mem=4G --pty bash -l

For a multi-core interactive shell:

$ srun --time=02:00:00 --mem=8G --cpus-per-task=4 --pty bash -l

For an interactive GPU session:

$ srun --time=02:00:00 --mem=8G --gres=gpu:1 --pty bash -l

Submitting Slurm preview batch jobs

Submit non-interactive Slurm preview jobs with sbatch. Slurm job scripts use #SBATCH directives in the script preamble.

A minimal Slurm batch script:

#!/bin/bash
#SBATCH --job-name=hello-slurm
#SBATCH --time=00:10:00
#SBATCH --mem=1G
#SBATCH --output=hello-slurm.%j.out

hostname
date

Submit the script with:

$ sbatch hello-slurm.sh

You can also pass Slurm resource options on the sbatch command line:

$ sbatch --time=02:00:00 --mem=4G hello-slurm.sh

Command-line options passed to sbatch take precedence over matching #SBATCH directives in the script.

Monitoring and canceling Slurm preview jobs

To list your Slurm preview jobs:

$ squeue -u $USER

To cancel a Slurm job:

$ scancel <JOB_ID>

where <JOB_ID> is the Slurm job ID reported by sbatch or squeue.

Production and Slurm preview documentation

Use Computing with Altair Grid Engine on the production cluster for the production Hoffman2 Cluster environment. The production page documents Altair Grid Engine concepts such as complexes, parallel environments, qrsh interactive sessions and qsub batch jobs.

Use this page for the Slurm preview cluster reached through login.hoffman2.oarc.ucla.edu. The Slurm preview page documents Slurm options and commands such as srun, salloc, sbatch, squeue and scancel, and notes the different operating system and application location under /u/local/apps.