Output layouts

A dynamic event tree writes one end-state file per sequence, three parameter-history files per sequence, one file per pivotal-event node, and three analysis logs per run. The volume of the pivotal-event class is set by the tree, not by the number of sequences, and that is why the engine offers a choice of layout for it.

--aggregate-pe selects the alternative layout. It is off by default.

Why pivotal-event output needs a choice of layout

Pivotal-event nodes are the structure of the tree, so there is one per branch point, and their number grows with both the width and the depth of the study. Two consequences follow that the other output classes do not share.

Their count dominates on long missions. Per sequence the engine writes one end-state file and three parameter-history files, and the tree adds one file per pivotal-event node. In the 12,306-sequence, 10,000-second studies/sgtr-10k tree (measured with the legacy-baseline-2026-08-18 engine, see Reference baselines) the rank directories held about 74 pivotal-event files per sequence, 94% of all files:

Output	Files per sequence	Share of files
ZPENode, pivotal-event nodes	about 74	94.3%
ZKPS / ZDIA / ZPIF, parameter histories	3	3.8%
ZESNode, end states	1	1.3%

Their count per sequence is not fixed. It rises with mission time, because a longer transient gives the crew model more opportunities to branch. The per-sequence file cost measured at one mission time cannot be reused to budget a longer study.

File-count quotas

A shared filesystem can limit the number of files as well as the bytes, and a large tree in the per-node layout reaches the file limit first. When a file-count quota is exhausted:

• a worker reports ERROR: write failed (disk full or I/O error): <path> on standard error and in its err messages.txt, while the filesystem still has free space (checked_close() in src/ADSIDAC/Global/CheckedIO.h);
• a merge in progress stops with a std::filesystem_error that carries the operating system's text, Disk quota exceeded.

On GPFS the limit can be a group quota. A user-scoped query does not show it; query the group and the device explicitly, with the device name that df prints:

mmlsquota -g <group> <device>

The remedy is to free files in the group's allocation (keep one archive per finished run instead of its file tree) and to run with --aggregate-pe. The merge mode matters too: under --merge=serial the rank-local files and their merged copies exist side by side until the end of the run, so the peak file count is about twice the final count; under --merge=now and --merge-only files are renamed, and the peak is about the final count.

What the aggregate layout changes

The aggregate layout writes the same pivotal-event content as length-delimited records in one file per worker rank (one file in a serial run), ZPENodes_agg.txt:

#PE <node_id> <content_bytes>
<exactly content_bytes of content>

The property that matters is what the file count becomes proportional to: rank count, which is fixed for a run, instead of node count, which is not. Output volume then tracks a number the analyst chooses, not the shape of the tree.

Three details of the format are deliberate:

• The header carries an explicit byte count, so the format is self-delimiting. Concatenating two aggregate files yields a valid aggregate file, which lets the MPI merge combine per-rank copies by appending them.
• Node ids may repeat, and the last occurrence wins. A node is appended again whenever its content changes (workers re-save a lineage after every sequence), and the merge resolves the repeats.
• Every renumbering path shares one implementation (finalize_sequential() in src/ADSIDAC/Global/PEOutput.h, with collect() + rewrite_sequential() as its content path), used by the MPI merge, by --merge-only and by the end of a resumed serial run, so they cannot diverge on layout.

While a run is in flight the aggregate is Output/PE/ZPENodes_agg.txt (in a rank directory under MPI). The final renumbering writes Output/PE/PEFolder_0/ZPENodes_agg.txt with record headers numbered 0 .. N-1.

Choosing a layout

Use the default when anything reads ZPENode_<id>.txt directly, as the pe-plotter tooling and the clustering tools do. This is why it is the default.

Use --aggregate-pe when the study is large enough that file count is the binding resource: long missions, wide trees, or a shared filesystem with a file-count quota. While the run is in flight the aggregate uses more bytes than the per-node layout, because a changed node is appended again instead of being overwritten. The merged output has the same content either way.

The same re-save behaviour multiplies the record count: a rank's aggregate carries every content revision of every node it held, about 4.5 records per unique id on a large tree. The merge's resolution is last-wins, so each rank's aggregate is compacted first (compact_aggregate(): last record per id, written to a temporary file and renamed into place). Under --merge=now every worker compacts its own file in parallel before staging it; --merge-only compacts each source file before concatenating it. The single-threaded final resolution then reads the compacted volume.

mpirun -n 128 adsidac --project-dir=. --input-deck=case.i --aggregate-pe

Reading either layout

tests/scripts/pe_canonical.py reads both and prints one line per node:

python3 tests/scripts/pe_canonical.py Output/PE            # <id> <hash> per node
python3 tests/scripts/pe_canonical.py Output/PE | wc -l    # node count

Prefer it to find for counting: find -name 'ZPENode*' also matches ZPENodes_agg.txt, so it returns 1 rather than the node count.

Guarantees and how they are tested

The two layouts carry identical information; only the file count differs. The tests assert this on content, because the file count is what the feature changes:

• tests/scripts/validate_pe_aggregate.sh runs a case in both layouts and compares the canonical node listings, while requiring the end-state, parameter-history and summary counts to match their reference values in both.
• BV4_Serial_PEAggregate_Validate and BV4_MPI_PEAggregate_Validate run it under ctest.

Comparisons are made within an execution mode. Serial and MPI runs produce different pivotal-event bytes in either layout: under MPI the node bodies carry rank-offset ids (rank times 100000), and the final renumbering changes file names and aggregate record headers only, never node bodies.

Scope

Aggregation covers pivotal-event nodes only, which is where the file count concentrates. The parameter histories are three files per sequence, but they are time series, so they dominate bytes: the 12,306-sequence reference tree named above occupies about 376 GB, mostly in those files. They are not aggregated.

When the merge runs

The MPI merge is controlled by --merge=now|serial|defer.

• now (default) is rename-based and worker-parallel (Client::self_merge_outputs(), Server::merge_outputs()). At shutdown each worker renames its own end-state and history files into the shared tree; file names carry the global sequence id, so ranks never collide, and a rename moves metadata, not bytes, because rank_N/ and Output/ are on one filesystem (the code falls back to copy and remove when a rename fails). Each worker also stages its three analysis logs, its compacted pivotal-event aggregate and its err messages.txt as Output/<ES|PE>/parts/<name>.rank_N. Rank 0 then moves the per-node pivotal-event files itself, in ascending rank order: the same node file exists on several ranks in different states of completeness, and renaming over an existing file in rank order makes the highest rank the deterministic winner. Rank 0 concatenates the staged parts in rank order, rebuilds ES_summary.txt, and renumbers the pivotal events sequentially.
• serial makes rank 0 copy every file in one pass and renumber through collect() + rewrite_sequential(), which reads every node into memory. It produces the same merged output.
• defer ends the run at tree completion. The rank directories remain the only copy of the outputs and Output/MERGE_PENDING.txt records the rank count, the pivotal-event layout, the sequence count and the configuration fingerprint. --resume is refused while that file exists.

In the per-node layout the sequential renumbering is a journaled two-pass rename with no content read (ZPENode_<old>.txt to PEFolder_0/tmp_<new>, then to PEFolder_0/ZPENode_<new>.txt). The mapping is written to Output/PE/renumber.map before the first rename, and a finalize that was interrupted resumes from that journal when it is called again. The result is a single PEFolder_0/ with ids 0 .. N-1 and no negative-id sentinel files.

A --checkpoint-stop run always merges inline, whatever --merge says, because a resume reads that intermediate merge. That merge does not renumber: pivotal-event files keep their natural ids until the run that completes the tree.

--merge-only

adsidac --merge-only --project-dir=. --input-deck=<deck>.i [--merge-threads=N]

--merge-only consolidates the rank directories of a deferred or interrupted merge and exits (run_merge_only() in ADSIDAC.cpp, merge_project() in src/ADSIDAC/Global/OutputMerge.h). It starts no simulation and no MPI, and it refuses to run under mpirun, because MPI start-up recreates the rank directories it has to read. Its steps are: end-state and history moves, rank-ordered pivotal-event moves, concatenation of the append files (from the rank directories or from parts a --merge=now worker had staged), ES_summary.txt rebuild, pivotal-event renumbering, removal of MERGE_PENDING.txt.

Option	Default	Meaning
--merge-threads	0	Threads for the end-state and history moves, one rank directory per thread at a time. 0 (or less) means the smaller of 8 and the hardware thread count. Never more threads are started than there are rank directories. The other steps are single-threaded. Used by --merge-only only

The pivotal-event layout is taken from MERGE_PENDING.txt. Without that file (salvage of an interrupted merge) it is taken from --aggregate-pe on the --merge-only command line.

Every step consumes its sources, so a killed --merge-only can be run again, and a finished one reports that there is nothing to do. One limit applies to a rerun: an analysis log is truncated before the remaining rank sources are concatenated, so blocks that the interrupted pass had already concatenated are not kept. --merge-only refuses to merge into an Output/PE that already holds a finalized run while rank directories still hold outputs.

The merge prints its wall time:

Merged <N> sequences, <M> PE nodes into <project>/Output in <X.X> s (rename)

with (serial copy) under --merge=serial and (merge-only, <R> rank dirs, <T> threads) under --merge-only.

The merge tests are BV4_MPI_MergeDefer_Validate and BV4_MPI_MergeDefer_Agg_Validate (tests/scripts/validate_merge.sh). They use no fixed counts: a --merge=now run defines the tree, a --merge=defer run followed by --merge-only must reproduce it, and two independent --merge-only passes over identical rank data must give byte-identical canonical pivotal-event listings and ES_summary.txt files. The aggregate variant also covers checkpoint and resume under --aggregate-pe.

See Restart payload trimming for the transport-side counterpart, where restart images are trimmed so that transfer volume is set by what the receiver reads, not by lineage depth.