Running on a Cluster

This page describes how to run large dynamic event trees under a batch system. The job scripts in scripts/hazel/ are working examples for one Slurm cluster with a GPFS filesystem and Open MPI. Their partition names, module names, paths and quota device are specific to that site; the structure carries over.

Shape of a job

#!/bin/bash
#SBATCH --nodes=4
#SBATCH --ntasks=128
#SBATCH --mem=180G
#SBATCH --time=01:55:00

module load <your MPI module>
BIN=/path/to/build/bin/adsidac
RUN=/scratch/$USER/runs/mystudy_$SLURM_JOB_ID

# 1. One fresh project directory per job, copied from the study
mkdir -p $RUN && cp -r /path/to/study/. $RUN/ && cd $RUN
rm -rf Output restart_base.bin

# 2. Environment variables are read once per process: export and forward them
export ADS_RESTART_TRIM=1
mpirun -n $SLURM_NTASKS -x ADS_RESTART_TRIM $BIN \
  --project-dir=. --input-deck=<deck>.i --truncation-time=<T> \
  --compress-queue --aggregate-pe > run.log 2>&1
rc=$?

# 3. Judge the run by its exit code and its outputs, then clean up
ES=$(find Output/ES -name 'ZESNode*' | wc -l)
echo "rc=$rc ES=$ES"
[ $rc -eq 0 ] && rm -rf rank_*

Points that matter:

• One project directory per run. An MPI run deletes and recreates every rank_<N>/ directory of its project at startup, and two runs in the same project directory destroy each other's working files.
• Keep the path short. RELAP5 refuses an input deck path longer than 80 characters.
• Steam tables. Keep tpfh2o, tpfh2onew and tpfd2o in the project directory (or launch from the directory that holds them). Each rank copies them into its rank directory.
• Forward the environment. ADS_* variables are read by every rank. With Open MPI use mpirun -x NAME.
• rank_<N>/ is not deleted by the engine. After a successful merge it holds only logs and scratch files. Remove it in the job script to return the files to the quota.
• Probability cutoff and mission time come from Input/Scheduler/ZCtrlPar.txt; --truncation-time overrides the time.

Memory

Resident memory is a per-rank quantity. A worker holds one restart image for every branch it has generated that no other worker has picked up yet, so its memory follows the width of the tree, not the mission time.

Apply the levers in this order:

1. Request enough memory per node from the batch system (--mem), sized as ranks per node times the per-rank peak.
2. ADS_RESTART_TRIM=1: each held image shrinks from the whole lineage's chain of restart records to one record.
3. --compress-queue: held and transferred images are LZ4-compressed, about 3x on RELAP5 restart data.
4. --memory-limit=<MB>: a per-rank ceiling. Above it, new restart images are written to restart_*.r files in the rank directory instead of being held in memory. Set it below node memory divided by ranks per node.

The server (rank 0) holds metadata only. It needs no more memory for a wide tree than for a narrow one.

File count and capacity

Two separate limits apply on shared filesystems.

File count. Per sequence the engine writes 1 end-state file, 3 history files and one pivotal-event file per branch point. The last class grows with mission time and dominates on long missions. --aggregate-pe replaces it with one file per rank. Under --merge=now a run's peak file count is about its final count; under --merge=serial it is about twice that, because rank-local files and their merged copies coexist until the end.

When a file-count quota is exhausted, workers report

ERROR: write failed (disk full or I/O error): <path>

while the filesystem still has free space, and a merge in progress stops with Disk quota exceeded. On GPFS the quota may be a group quota that a user-scoped query does not show. Query the group and the device explicitly:

mmlsquota -g <group> <device>      # device name as printed by df

A job can check the headroom before it starts and refuse to run, as scripts/hazel/sgtr10k_cvfix_kps.sbatch does: it computes the free file count from mmlsquota, compares it with HEADROOM (default 200000), and exits with code 75 when there is not enough.

Capacity. Bytes are dominated by the per-sequence histories (ZKPS, ZDIA, ZPIF), which are not aggregated. The 12,306-sequence reference tree in Reference baselines occupies about 376 GB. Extract the channels you need and archive the rest (below).

Ending the run at tree completion

With the default --merge=now the allocation is held until the merge is done. To release the nodes as soon as the tree is complete, defer the merge and consolidate in a single process, on the same or a later allocation:

mpirun -n $SLURM_NTASKS $BIN --project-dir=. --input-deck=<deck>.i --merge=defer ...
$BIN --merge-only --project-dir=. --input-deck=<deck>.i --merge-threads=8

--merge-only must not be started with mpirun. It is idempotent, so a merge killed by the wall clock can simply be run again.

Chained slices with checkpoints

A tree that does not fit one allocation can run as a chain of jobs that share a project directory and hand over through a checkpoint:

mpirun -n $SLURM_NTASKS $BIN --project-dir=. --input-deck=<deck>.i \
  --truncation-time=<T> --compress-queue \
  --checkpoint-dir=$RUN/checkpoint --checkpoint-interval=<N> --checkpoint-stop \
  $RESUME > run_$SLURM_JOB_ID.log 2>&1

where RESUME is empty for the first slice and --resume=<newest checkpoint_seq_*.dat> afterwards. scripts/hazel/sgtr10k_chain.sbatch is a complete self-resubmitting example.

Rules:

• End a slice on a sequence count, not on a signal. --checkpoint-stop makes the interval the slice length. Open MPI's mpirun exits on SIGUSR1 instead of forwarding it, so a batch pre-timeout signal ends the job without a checkpoint.
• Size the slice as planned work plus one longest sequence. An MPI checkpoint is written only after every worker has gone idle, so it costs up to one longest sequence. A slice that reaches its wall limit during that drain writes nothing.
• Repeat --truncation-time on every slice.
• Poll for the checkpoint file before resubmitting; its rename can complete a few seconds after mpirun returns. Resubmit only when a new checkpoint exists.
• Keep the project directory, restart_base.bin included, unchanged between slices. The resume checks a digest of the input deck and Input/.
• Pivotal events are renumbered only when the last slice completes the tree.

Measuring a scaling run

Wall-clock speedup alone cannot tell a slow scheduler from a tree that has run out of work. Record the server's occupancy trace with an absolute path and summarize it:

mpirun ... --occupancy-log=$RUN/occ.csv
python3 scripts/occupancy_report.py $RUN/occ.csv

See Measuring occupancy for the columns and the interpretation.

Extracting channels and archiving

scripts/hazel/kps_extract.sh cuts the ZKPS histories of a finished run down to named columns and packs them with ES_summary.txt and Scenario info.txt:

kps_extract.sh <run_dir> <out.tgz> [Time,Col1,Col2,...]

The column list defaults to 15 channels used by the clustering studies; pass your own list, starting with Time. The extracted files keep the header layout of a ZKPS file.

Variable	Default	Meaning
KPSX_MODE	direct	direct streams files into the archive through kps_direct.py; scratch writes the cut files to a scratch directory first
KPSX_JOBS	the job's CPU count, at most 128	Parallel workers
KPSX_COMPRESS	gzip	gzip or zstd (with zstd the output name becomes .tar.zst)
KPSX_CUT	awk when available	Column cutter used by kps_direct.py: awk or py

To return a finished run's files to the quota, pack the run directory into one archive, verify the archive (for example by counting its ZESNode members), and only then delete the directory. scripts/hazel/archive_runs.sbatch does this for a list of run directories.

Script index (scripts/hazel/)

Script	Purpose
sgtr10k_smoke.sbatch	Short correctness run of the SGTR deck
sgtr10k_longsmoke.sbatch	Full-length single run sized for backfill
sgtr10k_growth.sbatch	One tree-growth measurement at a given T and CUTOFF; writes a CSV row and deletes its run directory
sgtr10k_scale.sbatch, sgtr10k_submit_ladder.sh	One strong-scaling rung, and the driver that submits the ladder (--go submits; without it, a dry run)
sgtr10k_t10k*.sbatch	The full-length tree in one allocation, one variant per partition and rank count
sgtr10k_chain.sbatch	Self-resubmitting chain of checkpointed slices
sgtr10k_kps.sbatch, sgtr10k_cvfix_kps.sbatch	Runs that keep their outputs and extract channels; the second adds the quota check, the branch floor variables, MERGE=defer and EXTRACT=0
sgtr1k_viz.sbatch	Run with the visualization server enabled
kps_only.sbatch, kps_extract.sh, kps_direct.py	Channel extraction
merge_lite.sbatch	Shell consolidation of rank directories without pivotal-event renumbering; adsidac --merge-only does the complete job
archive_runs.sbatch	Pack, verify and remove run directories
build_cvfix.sbatch, report_status.sbatch	Build a second engine tree; write a status summary