Computing overview

What is a compute cluster? What is a node?

A compute cluster is made of individual nodes which are interconnected between themselves and which aggregate power can be harnessed to address problems which nature requires either distributed or capacity computing.

A node, or host, is a physical server comprised of multiple CPU cores (which are referred to as slots in the scheduler lingo) rack-mounted in one of our two data centers and interconnected to other nodes (as well as to the storage) to form the cluster.

Upon connecting/logging into the cluster users access the cluster via its login nodes. Login nodes are special hosts which sole purpose is to provide a gateway to the compute nodes and their computational resources. For more information, see Role of login nodes.

Computational resources (such as memory, cores, runtime, CPU-type, GPU-type, etc.) on board of compute nodes are managed by a job scheduler. Any CPU, GPU or memory intensive computing task should be performed on compute nodes through the scheduler that applies to the environment where you are working.

The Computing section has three main pages:

• this overview page, which explains node types, group-owned nodes, and the meaning of highp, shared and campus jobs

• Computing with Slurm, for the Hoffman2 Cluster reached at hoffman2.oarc.ucla.edu, which uses the Slurm workload manager and commands such as srun, salloc, sbatch, squeue and scancel

• Computing with Altair Grid Engine on the legacy cluster, for the legacy cluster reached at hoffman2.idre.ucla.edu, which uses Altair Grid Engine commands such as qrsh, qsub, qstat and qdel

Grid Engine commands and job scripts do not work with Slurm. To convert your Grid Engine job scripts, see Converting Grid Engine job scripts to Slurm.

Node types and access

• Node types

• Group-owned nodes

• Highp vs shared vs campus jobs

Node types

A summary of the types of nodes that you will encounter while using the Hoffman2 Cluster and a description of their intended use is given in the Types of nodes on the Hoffman2 Cluster table:

Types of nodes on the Hoffman2 Cluster

login nodes	you access a login node upon connecting to the Hoffman2 Cluster via: terminal and SSH at the fully qualified domain name: hoffman2.oarc.ucla.edu (legacy cluster: hoffman2.idre.ucla.edu) remote desktop (legacy cluster only, for now) at the NoMachine or X2Go services running on: nx.hoffman2.idre.ucla.edu or x2go.hoffman2.idre.ucla.edu respectively Note Login nodes are meant for light-weight tasks such as editing your code and submitting jobs to the scheduler. Login nodes are a resource shared by many concurrent users and are not intended for CPU or memory intensive tasks. Please see Role of login nodes. Important All CPU and/or memory intensive (as well as GPU) computations need to run on compute nodes accessed via the scheduler. See Computing with Slurm (for the legacy cluster: Computing with Altair Grid Engine on the legacy cluster).
Data transfer nodes	The Hoffman2 Cluster has two dedicated and performance-tuned data transfer nodes with advanced parallel transfer tools to support your research workflows.
CPU-based compute nodes	Most of the nodes on the Hoffman2 Cluster are CPU-based compute nodes. These are where your jobs execute after they are scheduled through Slurm (or, on the legacy cluster, Altair Grid Engine).
GPU-based compute nodes	A portion of compute nodes on the Hoffman2 Cluster is equipped with one or more GPU cards available on the Hoffman2 Cluster of various types. Please refer to: Role of GPU nodes to see what workload is best suited to run on these nodes. To request GPUs, see Requesting GPUs with Slurm (legacy cluster: GPU access).

The Hoffman2 Cluster has a number of compute nodes available to the entire UCLA community. Additionally, research groups can purchase dedicated compute nodes. Users in groups who have contributed nodes to the cluster can access their nodes in a preferential fashion and for extended runtimes or access unused cores across the wider cluster (see: Highp vs shared vs campus jobs).

Group-owned nodes

Group-owned nodes, allow users to run jobs (interactive or batch) on their computational resources for an extended runtime (up to fourteen days). Moreover, the portion of the jobs submitted to owned-resources, that can be concurrently allocated on them, are guaranteed to start within twenty-four hours from their submission (wait time is typically less). Node ownership also allow users in that group to access any currently unused resource owned by a different group for up to a runtime of 24 hours.

If your group is interested in purchasing nodes, please visit: Purchasing additional resources.

Highp vs shared vs campus jobs

In the Hoffman2 Cluster jargon jobs submitted to owned resources are referred to as highp jobs while jobs submitted to the currently unused resources of any contributing group, including the submitter’s own group, are referred to as shared jobs. Jobs submitted by users in groups that have not purchased nodes are limited to run on OARC-owned resources for up to 24 hours; jobs from these users are referred to as campus jobs and the users as campus users.

With Slurm, campus and shared jobs are selected automatically from your Slurm account; to run a highp job, add --qos=highp. See Accounts, QOS and where your jobs run.

Note

Highp and shared jobs are only for users in groups that have contributed nodes to the cluster.

See also: Job scheduling policy.