How a Linux HPC Cluster Runs a Job - Yenra

The job moves through several distinct services

1. Prepare: use a login or development node according to site policy to edit files, transfer inputs, and select the software environment. Heavy computation belongs in an allocation.
2. Queue: submit a request describing time and resources. The scheduler considers available resources and site policies; an accepted request can remain pending.
3. Allocate: compute nodes are assigned to the job. A node is a machine; a task is normally a process launched within the allocation.
4. Run: the launcher starts the requested work. Processes may communicate across the interconnect and read or write storage.
5. Validate: check exit states and scientific or application results, retain useful outputs, then release temporary storage according to policy.

The Slurm user quick start explains jobs, partitions, and basic commands. Slurm's sbatch manual notes that submitting the script does not automatically copy other input files to the compute nodes. A shared path or an explicit staging procedure is necessary.

A shared filesystem lets nodes access common files but can become a bottleneck when many tasks create tiny files or repeatedly scan the same directories. Node-local scratch may be faster for temporary work, but its contents and lifetime are site-specific. Neither scratch nor a shared project directory should be assumed to be backed up.

Choose a parallel model the application actually supports

On a small screen, scroll the table sideways to read all columns.

Four tasks with one CPU each differ from one task with four CPUs. The latter does not make a serial program threaded. CPUs in Slurm are configured allocation units and may correspond to cores or hardware threads. Use the site's terminology and placement guidance. See the srun manual for task launch and binding options; do not copy an MPI plugin choice from another cluster without checking it.

A small, inspectable two-node job

Download the Slurm script, Python task, result checker, and notes (ZIP). The task divides the integers 1 through 10,000 among four ranks and sums their squares. Each process reads its Slurm rank and prints one JSON record. The pieces are independent and use no MPI library.

Place the extracted files in a directory accessible at the same path on the compute nodes. Confirm that Python 3 is available there; a site may require its documented environment-module setup. Add the account, partition, or quality-of-service settings required by your site before submission.

#!/bin/bash
#SBATCH --job-name=yenra-learning
#SBATCH --nodes=2
#SBATCH --ntasks=4
#SBATCH --ntasks-per-node=2
#SBATCH --cpus-per-task=1
#SBATCH --mem=1G
#SBATCH --time=00:02:00
#SBATCH --output=yenra-%j.out
#SBATCH --error=yenra-%j.err

set -euo pipefail
cd "$SLURM_SUBMIT_DIR"
srun --kill-on-bad-exit=1 python3 partial_sum.py

The request is two nodes, four processes in total, two processes per node, one CPU per process, and 1G of memory per node. The time limit is two minutes of allocated runtime, not a promise about queue delay. The %j token inserts the job ID in output filenames. All directives precede the executable shell commands because Slurm stops reading directives at the first command.

These resources are intentionally oversized for the tiny calculation so node placement is visible. Use a training allocation approved by the site. The Python computation and checker were tested locally, and Bash syntax was checked; this example has not been executed on a live Slurm cluster. Local tests cannot verify a site's scheduling, filesystem, or environment configuration.

Submission success is not result validation

From the shared example directory, submit sbatch learning.sbatch. Record the returned job ID. Use squeue -j JOBID while it is pending or running, replacing JOBID with that number. A pending reason helps distinguish policy limits from unavailable resources; requesting a longer runtime can make placement harder.

When the job finishes, inspect the error file and run python3 check_results.py yenra-JOBID.out using its actual output filename. Records can arrive in any rank order. The checker requires all four ranks exactly once, checks each assigned interval and partial sum, and verifies the combined total:

Verified 4 ranks; sum of squares 1..10000 = 333383335000

Inspect accounting with sacct -j JOBID --format=JobID,State,ExitCode,Elapsed,AllocCPUS where accounting is enabled. The sacct reference explains job and step records. Look at failed steps as well as the batch job. A zero exit code still does not prove a scientific result is correct; the separate numerical check provides that evidence for this particular example.

Compare useful speedup with resource cost

Invented benchmark: a different, realistic fixed-size workload produces the same validated result in each run. The following elapsed times are fictional; they are not measurements from the teaching script.

On a small screen, scroll the table sideways to read all columns.

Speedup is 120 divided by elapsed minutes; efficiency is speedup divided by node count. Moving from four to eight nodes saves 9 minutes, or 23.1% of the four-node elapsed time. Allocated node-minutes rise from 156 to 240, a 53.8% increase. These are allocation-time units, not measured energy or the site's billing formula.

Measure repeated runs, result correctness, queue delay, communication, and input/output costs before choosing a size. Strong scaling keeps the problem fixed; weak scaling increases the problem with the resources and answers a different question. The appropriate choice may be the faster deadline, greater total throughput, or lower allocation use.