Running ADS-IDAC

Basic Usage

Serial mode

./adsidac --project-dir=<path> --input-deck=<path_to_.i_file>

MPI parallel mode

mpirun -n <nprocs> ./adsidac --project-dir=<path> --input-deck=<path_to_.i_file>

The engine detects an MPI launch from the OMPI_COMM_WORLD_SIZE or PMI_SIZE environment variable. Rank 0 is the server and schedules the tree; ranks 1 to N-1 are workers and run sequences, so -n 2 gives one worker.

Working directory and steam tables

RELAP5 reads the steam tables (tpfh2o, tpfh2onew, tpfd2o) from its working directory.

• Serial: the directory you launch from must contain the steam tables. The build copies them to build/bin/, and the packaged wrapper script links them into the current directory before it starts the binary.
• MPI: each rank works in <project-dir>/rank_<N>/. At startup the engine copies the steam tables into every rank directory, taking them from the launch directory if they are there and from the project directory otherwise. Both cd build/bin && mpirun ... --project-dir=<p> and cd <p> && mpirun /abs/path/adsidac --project-dir=. work.

--project-dir and --input-deck may be relative; both are made absolute when the options are parsed. RELAP5 refuses an input deck path longer than 80 characters, so keep project directories shallow.

Example with the BV4 test case

From build/bin/ (the build copies tests/ into build/tests/):

# Serial
./adsidac --project-dir=../tests/BV4 --input-deck=../tests/BV4/BV4.i

# MPI with 4 ranks (1 server + 3 workers)
mpirun -n 4 ./adsidac --project-dir=../tests/BV4 --input-deck=../tests/BV4/BV4.i

The expected output counts are listed in Reference baselines.

tests/BV3 is reference-only

tests/BV3/ does not run on this engine: its input files use formats the current parsers do not accept. Use tests/BV4/ as the working reference case.

CLI Reference

Option	Default	Description
--help, -h		Print the option list and exit
--version, -v		Print the version and exit
--info, -i		Print name, version and maintainer and exit
--project-dir	required	Project directory containing Input/ and receiving Output/
--input-deck	required	RELAP5 input deck (.i file)
--truncation-time	0	Override Truncation_time from ZCtrlPar.txt, in seconds (0 = use the file value). Must be repeated on resume commands
--enable-rm-cycle	false	Enable the ReasonMachine snapshot, restore and serialize cycle (developer feature; see the ReasonMachine internals page)

--help, --version and --info are handled before the required options are checked and before MPI starts, so they work without a project.

Solver options

Option	Default	Description
--solver	relap	Solver backend: relap or 0 (RELAP5), mock or 1 (REST API mock solver)
--solver-port	9090	Mock solver port on localhost
--solver-url		Full mock solver URL (for example http://remote-host:8080). Overrides --solver-port. Use it together with --solver=mock

Performance options

Option	Default	Description
--no-thermo-cache	false	Disable thermodynamic table caching (reload the tables from file every sequence)
--no-memory-restart	false	Disable in-memory restart (write and read restart files instead)
--memory-limit	0	Resident-memory limit in MB for adaptive restart (0 = unlimited). Above the limit, new restart images go to files instead of memory, decided per branch point
--compress-queue	false	LZ4-compress restart images held and transferred by MPI workers
--occupancy-log		MPI only. CSV file that records queue depth and busy workers on every change (see Measuring occupancy)

Choosing performance options

The defaults (thermodynamic table cache on, in-memory restart on) are the fastest configuration and need no tuning. Deviate only for a reason:

• --no-memory-restart: choose it when RAM is the binding constraint and the disk is fast. In-memory restart keeps one restart image per pending branch point, so resident memory grows with the width of the tree. File mode writes one restart_*.r file per branch point into the working directory (the rank directories under MPI). They are removed at shutdown but accumulate while the run is in flight, so budget scratch space for them.
• --memory-limit <MB>: the middle ground. The run stays in memory until the process exceeds the limit, then writes new restart images to files. Use it on shared nodes with a known RAM budget, for example --memory-limit 4096. The limit applies per process, so under MPI it is a per-rank figure. It cannot be combined with --no-memory-restart.
• --compress-queue: spends CPU to shrink restart images (about 3x on RELAP5 restart data). It pays when memory or the interconnect is the constraint and costs wall time on short missions on a single fast node, where moving the image through shared memory is cheaper than compressing it. Measure your machine with scripts/mpi_transport_probe.sh (below) when unsure.
• ADS_RESTART_TRIM=1: see Environment variables. It is the setting that keeps restart transfers at a constant size on deep trees.
• Rank count: a run uses 1 server and N-1 workers. The useful number of workers is bounded by the cores available and by the width of the tree: once every pending branch has a worker, more ranks stay idle. The occupancy log tells the two cases apart.

Small structured payloads (the crew state exchanged at every branch point) are always LZ4-compressed on the wire. Streams above 4 MB, which carry the thermal-hydraulic restart image, are sent uncompressed unless --compress-queue is set.

Measuring your transport (scripts/mpi_transport_probe.sh)

Before tuning compression on a new machine or cluster, run the probe. It measures the MPI bandwidth between two ranks and the LZ4 compress and decompress throughput on restart-like data, then prints a verdict per payload size with the break-even bandwidth:

./scripts/mpi_transport_probe.sh                                  # default transport
PROBE_MPIRUN_FLAGS="--mca btl self,tcp" ./scripts/mpi_transport_probe.sh   # force TCP

Compression pays only when the measured bandwidth is below a row's break-even.

Measuring occupancy

mpirun -n <nprocs> ./adsidac --project-dir=<path> --input-deck=<deck>.i \
  --occupancy-log=$PWD/occ.csv
python3 scripts/occupancy_report.py occ.csv

The server writes one CSV row each time the queue depth or the number of busy workers changes. The state is a step function, so logging changes loses nothing. Columns:

elapsed_s,queue_depth,busy_workers,total_workers,sequences_completed

Give an absolute path. The server opens the file after it has moved into its rank directory, so a relative path lands in <project-dir>/rank_0/, which the next MPI run in that project deletes. The file is truncated at the start of the run. The option has no effect on the simulation outputs and is ignored in serial mode.

scripts/occupancy_report.py integrates the step function over time and prints, per file: the number of workers and the time span, the mean number of busy workers, the time-weighted occupancy, the mean queue depth, the share of wall time with every worker busy, and the share of wall time starved (at least one idle worker while the queue is empty). Low occupancy with an empty queue means the tree had no more work to hand out, so the run measured the input deck rather than the scheduler. Low occupancy with a deep queue points at dispatch.

Output layout options

Option	Default	Description
--aggregate-pe	false	Write pivotal-event nodes as records in one file per rank instead of one file per node
--merge	now	MPI output merge mode: now (rename-based, worker-parallel), serial (rank-0 copy loop), defer (end the run at tree completion; consolidate later)
--merge-only	false	Consolidate the per-rank outputs of a deferred or interrupted merge into Output/ and exit. Run it without mpirun
--merge-threads	0	Thread pool size for the --merge-only rank-directory moves (0 = the smaller of 8 and the hardware thread count)

Merge modes. Under --merge=now each worker renames its own end-state and history files into the shared Output/ tree at shutdown (a metadata operation, no bytes are copied) and stages its logs as ordered parts. Rank 0 then concatenates the parts in rank order, rebuilds ES_summary.txt, moves the pivotal-event files, and renumbers the pivotal events sequentially. The run prints one line when it is done:

Merged <N> sequences, <M> PE nodes into <project>/Output in <X.X> s (rename)

--merge=serial makes rank 0 copy every file itself, in one process. It produces the same merged output, except that the workers' err messages.txt files stay in their rank directories. Because it copies instead of renaming, the rank-local files and their copies exist side by side until the run ends, so the peak file count is about twice that of --merge=now.

--merge=defer ends the run when the tree ends. The rank directories remain the only copy of the outputs, and Output/MERGE_PENDING.txt records the rank count, the pivotal-event layout, the sequence count and the configuration fingerprint. Consolidate later, on any allocation, with

./adsidac --merge-only --project-dir=<path> --input-deck=<deck>.i

--merge-only starts no simulation and no MPI. It moves the rank directories' files with a thread pool (--merge-threads, never more threads than rank directories), then performs the same concatenation, summary rebuild and renumbering as the inline merge. It is idempotent: a killed merge resumes where it stopped, consuming both rank-local files and any parts a --merge=now worker had already staged, and a finished merge reports that there is nothing to do. One limit applies: the three analysis logs (Scenario info.txt, Procedure data.txt, Scan data.txt) are rebuilt from the sources that are still present, so a merge killed during the log concatenation itself leaves those logs without the blocks of the ranks it had already consumed. End states, histories and pivotal events are not affected. Without a MERGE_PENDING.txt it merges whatever the rank directories hold, taking the pivotal-event layout from --aggregate-pe on its own command line.

Rules that follow from this design:

• --resume is refused while Output/MERGE_PENDING.txt exists, because starting an MPI run recreates the rank directories that hold the unmerged outputs. Run --merge-only first.
• A --checkpoint-stop run always merges inline, whatever --merge says, because a resume reads that intermediate merge.
• --merge-only refuses to run under mpirun, and it must be given on the command line (not in ZiniADS.txt).
• --merge-only cannot be combined with --resume or --checkpoint-interval.
• Rank directories are not deleted at the end of a run. After a merge they hold only logs (rank_output.log, rank_errors.log) and scratch files. The next MPI run in the same project deletes and recreates them.

Pivotal-event layouts. A dynamic event tree writes one pivotal-event file per branch point, so this output class scales with the tree, and on long missions it dominates the file count: in the 12,306-sequence reference tree listed in Reference baselines it is 94% of all files, about 74 per sequence against 3 parameter-history files and 1 end-state file.

--aggregate-pe writes the same content as length-delimited records in one file per rank, which makes the pivotal-event file count depend on the rank count (fixed for a run) instead of the node count:

mpirun -n 128 ./adsidac --project-dir=. --input-deck=case.i --aggregate-pe

Keep the default when something reads ZPENode_<id>.txt directly, as the pe-plotter tooling does. Choose --aggregate-pe when the file count is the binding resource: long missions, wide trees, or a filesystem with a file-count quota. While the run is in flight a node is appended to the aggregate again each time its content changes, so the aggregate uses more bytes than the per-file layout; the merge keeps the last record of each node, and the merged output has the same content either way.

The two layouts carry identical information. The BV4_Serial_PEAggregate_Validate and BV4_MPI_PEAggregate_Validate tests compare node content between them. To count nodes in either layout use

python3 tests/scripts/pe_canonical.py Output/PE | wc -l

rather than find -name 'ZPENode*', which also matches ZPENodes_agg.txt.

Capacity, as distinct from file count, is driven by the per-sequence parameter histories (ZKPS, ZDIA, ZPIF), which are not aggregated. See Output layouts and Running on a cluster.

Checkpoint and resume options

Option	Default	Description
--checkpoint-dir		Directory for checkpoint files (created if missing)
--checkpoint-interval	0	Write a checkpoint every N completed sequences (0 = disabled). Requires --checkpoint-dir
--checkpoint-stop	false	Exit after writing a checkpoint. Requires --checkpoint-interval
--resume		Resume from a checkpoint file

# Checkpoint during a serial run and stop
./adsidac --project-dir=<path> --input-deck=<deck>.i \
  --checkpoint-dir=./ckpt --checkpoint-interval=5 --checkpoint-stop

# Resume under MPI
mpirun -n 4 ./adsidac --project-dir=<path> --input-deck=<deck>.i \
  --resume=./ckpt/checkpoint_seq_5.dat

Checkpoint files are named checkpoint_seq_<completed sequences>.dat, are LZ4-compressed, and are written to a temporary name and renamed when complete. Every save prints [CHECKPOINT] Saved: <path> (<N> sequences, <M> pending) followed by the matching --resume= argument.

Supported combinations. A checkpoint written by a serial run resumes in serial or under MPI. A checkpoint written by an MPI run resumes under MPI with the same or a different number of ranks, in memory or file restart mode, with or without --compress-queue.

MPI checkpoint to serial resume is not supported

Resuming a checkpoint written by an MPI run without mpirun does not terminate: after the pending branches from the checkpoint are done, new sequences end immediately with "terminated for unknown reason" and the run keeps producing them. Resume MPI checkpoints under MPI (mpirun -n 2 gives a single worker).

Triggers. Besides the interval, sending SIGUSR1 to the adsidac process (rank 0 under MPI; workers ignore the signal) requests a checkpoint at the next opportunity, provided --checkpoint-dir is set. The signal has to reach that process itself. Open MPI's mpirun does not forward SIGUSR1 and exits when it receives it (exit code 138), so a batch system's pre-timeout signal delivered to the launcher ends the job without a checkpoint. Under a batch system, end a slice on a sequence count with --checkpoint-interval and --checkpoint-stop instead; see Running on a cluster.

Drain under MPI. An MPI checkpoint is written only when every worker is idle. Once a checkpoint is pending the server stops handing out work and waits for the running sequences to finish, so a checkpoint costs up to one longest sequence of wall time. A batch slice that reaches its wall limit during the drain writes nothing. Size chained slices as the planned work plus one longest sequence.

Resume rules

• Repeat the same --truncation-time on the resume command. The checkpoint does not carry the command-line override. A mismatch produces no error; the resumed lineages simply end at a different time from the ones already completed.
• Wait for checkpoint_seq_*.dat to appear after a --checkpoint-stop run returns. The write and rename of a large file can complete a few seconds after mpirun exits. Scripts should poll for the file.
• The configuration is fingerprinted. Every run digests the input deck, ZCtrlPar.txt and every file under Input/, and prints [CONFIG] fingerprint=0x... at startup. A resume is refused if the digest differs from the one in the checkpoint. ZiniADS.txt, the steam tables, Output/ and command-line overrides are not part of the digest.
• A finalized output directory refuses resumes. Once a run has completed and renumbered its pivotal events, resuming into that Output/ is refused. Move Output/ aside to rerun from the checkpoint.
• Pivotal-event names at a checkpoint stop. A --checkpoint-stop run leaves pivotal-event files under their rank-window names; the sequential renumbering happens only when the tree completes.
• Checkpoint files are version-gated. The loader accepts format version 9 only and refuses older files. There is no cross-version migration.
• --compress-queue follows the checkpoint. If the resume command disagrees with the setting the checkpoint was written with, the engine switches to the checkpoint's setting and logs the override.
• With ADS_RESTART_TRIM=1, keep restart_base.bin. The checkpoint holds trimmed restart images, and the resume reads <project-dir>/restart_base.bin to complete them.
• Disk budget. A checkpoint holds one restart image per pending branch, so its size follows the width of the tree at the moment of the save.

Visualization options (MPI only)

Option	Default	Description
--viz-port	0	Port of the real-time DET visualization web server on rank 0 (0 = disabled)
--viz-outputs		Output types streamed to the server: events, es, all (comma-separated). Requires --viz-port

mpirun -n 4 ./adsidac --project-dir=<path> --input-deck=<deck>.i --viz-port=8888

# With output streaming
mpirun -n 4 ./adsidac --project-dir=<path> --input-deck=<deck>.i \
  --viz-port=8888 --viz-outputs=events,es

Then open http://localhost:8888 in a browser:

• /: DET tree (swimlane layout, branch folding, ancestry highlighting)
• /pe: pivotal-event graph (live node graph, color-coded by event type)
• /api/tree: JSON tree state
• /api/events: pivotal events per sequence
• /api/end-states: end states per sequence

Notification options

Option	Default	Description
--notify-ntfy		Send an ntfy.sh notification on completion. Without a value the topic is generated from the hostname and printed at startup; with a value that topic is used
--notify-discord		Send a completion notification to a Discord webhook URL

./adsidac --project-dir=<path> --input-deck=<deck>.i --notify-ntfy
./adsidac --project-dir=<path> --input-deck=<deck>.i --notify-ntfy=my-topic
./adsidac --project-dir=<path> --input-deck=<deck>.i --notify-discord=<webhook-url>

Options that apply to one mode only

--viz-port, --viz-outputs, --occupancy-log, --merge and --compress-queue act only under MPI. A serial run accepts them and ignores them. --merge-only is the reverse: it runs serially and refuses an MPI launch.

Option checks

The engine validates the option set before it starts and exits with code 1 and one of these messages:

Condition	Message
Unknown or malformed option	ERROR: <detail> followed by Check ./adsidac --help
--solver-url with --solver=relap	ERROR: --solver-url cannot be used with --solver=relap
--solver value other than relap, 0, mock, 1	ERROR: unknown solver '<x>'. Use relap|0 or mock|1
--no-memory-restart with an explicit --memory-limit above 0	ERROR: --no-memory-restart and --memory-limit are mutually exclusive
--viz-outputs without --viz-port	ERROR: --viz-outputs requires --viz-port to be set
--viz-outputs token other than events, es, all	ERROR: unknown viz output type '<token>'
--checkpoint-interval without --checkpoint-dir	ERROR: --checkpoint-interval requires --checkpoint-dir
--checkpoint-stop without --checkpoint-interval	ERROR: --checkpoint-stop requires --checkpoint-interval
--merge value other than now, serial, defer	ERROR: unknown merge mode '<x>'. Use now, serial, or defer
--merge-only with --resume or a checkpoint interval	ERROR: --merge-only does not run a simulation; it cannot be combined with --resume or checkpointing
--merge-only under mpirun	ERROR: --merge-only runs serially; launch it without mpirun
--resume while Output/MERGE_PENDING.txt exists	ERROR: this project has a deferred, unmerged run (Output/MERGE_PENDING.txt). Run --merge-only first; ...

Options file (ZiniADS.txt)

If a file named ZiniADS.txt exists in the launch directory, it is read as an options file with the same option names as the command line, one name=value per line. Values given on the command line take precedence. --help, --version, --info and --merge-only are recognized on the command line only.

Environment variables

Each variable is read once per process, so it must be set before launch. Under Open MPI pass it to every rank with mpirun -x NAME.

Variable	Default	Effect
ADS_RESTART_TRIM	off	1 enables restart trimming (MPI only), described below
ADS_BRANCH_FLOOR	0 (off)	Minimum probability of a branch alternative, described below
ADS_BRANCH_FLOOR_SINGLE	value of ADS_BRANCH_FLOOR	Threshold for forks with a single explicit alternative, described below
ADS_RESTART_INJECT	on	0 makes a restarted child sequence start without the content of its own branch alternative. It reproduces the trees of the legacy-baseline-2026-08-18 tag (see Reference baselines); leave it unset for analysis runs

ADS_RESTART_TRIM=1. A RELAP5 restart image is an append-only chain of restart records, and a child sequence inherits its parent's whole chain, so untrimmed transfers grow with the depth of the lineage (from a few MB to more than 100 MB). With trimming a worker ships only the one record the receiver restores, a few MB regardless of depth. The part common to all images of a run (the header and the first complete record) is written once to <project-dir>/restart_base.bin and spliced back by the receiver. If the record cannot be located the worker sends the whole image and logs [TRIM] ... not found in buffer. Set it for long missions and wide trees. Two rules follow:

• restart_base.bin belongs to the run's inputs. It is created only if it does not exist, and the engine does not delete it. Delete it whenever the input deck or the plant model changes, and keep it next to a checkpoint you intend to resume.
• Serial runs ignore the variable.

Design details are in Restart payload trimming.

ADS_BRANCH_FLOOR=<p>. Applied at every branch point, to each fork separately, before the forks are combined into the set of child sequences:

• In a fork with several alternatives, every alternative other than the first whose probability is below p is removed, and its probability is added to the first alternative (the one the running sequence itself continues with). The fork still sums to the same total. A fork reduced to one alternative opens no child sequence.
• A fork with a single explicit alternative of probability q < 1 has an implicit complement 1 - q that is not simulated. If that complement is below ADS_BRANCH_FLOOR_SINGLE, q is set to 1.

The floor bounds tree growth by not opening children that the probability cutoff in ZCtrlPar.txt would cut shortly afterwards, and it keeps their probability in the tree instead of dropping it. ADS_BRANCH_FLOOR_SINGLE has no effect unless ADS_BRANCH_FLOOR is above 0.

Diagnostic variables

These print additional information and do not change results. They are meant for development.

Variable	Set to	Output
ADS_MEMBUF_TRACE	1	[MEMBUF] restart-buffer capture statistics and [TRIM] lines with the size before and after trimming
ADSIDAC_BRANCH_DIAG	any value	[BRANCH-DIAG] and [SEQ-FP] lines on stderr for every branch point and sequence
ADSIDAC_OPERATOR_PROFILE	any value	Serialized size of each member of an operator snapshot
ADSIDAC_SNAPSHOT_PROFILE	any value	Copy time and serialized size of each member of a restart snapshot

ADS_MEMBUF_TRIM=1 selects an experimental restart-buffer capture that keeps only the last restart record. It is not sufficient for a restart and is not for analysis runs.

Interactive Interrupt

Press Ctrl+C during a run to get a menu:

1. Continue simulation (default)
2. Save checkpoint and exit, written to --checkpoint-dir, or to <project-dir>/checkpoint/ when that option is not set
3. Exit without saving

It works in serial and MPI mode and costs nothing when not used.

Mock Solver

The mock thermal-hydraulic solver allows full dynamic event tree runs without RELAP5 or any Fortran dependency. It uses a simplified lumped-parameter PWR model with 310 output parameters.

# Build the mock solver (a separate CMake project, no Fortran needed)
cmake -S mock-solver -B mock-solver/build -G Ninja -DCMAKE_BUILD_TYPE=Release
cmake --build mock-solver/build

# Start it, then run adsidac against it (from the repository root)
./mock-solver/build/bin/mock-solver --port 9090 &
cd build/bin && ./adsidac --project-dir=../tests/BV4 --input-deck=../tests/BV4/BV4.i --solver=mock

# MPI with the mock solver
mpirun --oversubscribe -n 4 ./adsidac --project-dir=../tests/BV4 \
  --input-deck=../tests/BV4/BV4.i --solver=mock

See Mock Solver Architecture.