Restart payload trimming

In an MPI run every branch point hands a RELAP5 restart image from the worker that generated it to the workers that simulate its branches. Restart payload trimming reduces that image to the one restart record the receiver restores from. It is off by default and enabled with the environment variable ADS_RESTART_TRIM=1. The user-facing summary is in Running a simulation; this page describes the mechanism.

The restart image

The engine always runs RELAP5 with its restart-plot unit redirected into one append-only byte buffer (src/Relap/restart_membuf.f90). --no-memory-restart and --memory-limit only decide whether a captured copy of that buffer is kept in memory or written to a file. The buffer starts at 64 MiB and doubles when it is full. Every record in it is framed the same way, with little-endian 32-bit lengths:

[reclen][payload: reclen bytes][reclen]      repeated

so the buffer can be walked record by record without knowing what the records mean.

The image starts with the file header and the plot-identification block, which are written once. After them comes a chain of restart records ("dumps"). rstrec.f appends one each time RELAP5 writes a restart, which dtstep.f does at every branch point that generates more than one branch. A dump opens with an 8-byte header record followed by a 24-byte record with this payload:

Bytes	Content
0 to 7	the text restart
8 to 11	flag word; bit 4 (value 16) marks a complete restart
16 to 19	ncount, the restart number that card 103 of indeck.i selects

Two properties of the chain matter here:

• A dump is either a complete restart or a smaller incremental one. The chain opens with a complete restart (flag bit 4 set) and the incremental dumps that follow depend on it. On the BV4 three-loop PWR deck the header block is 0.25 MB, the opening complete dump 4.86 MB and an incremental dump 2.34 MB.
• A child inherits its parent's chain. A restored sequence continues to append to the image it was restored from, so an untrimmed image holds one dump for every branch point of the whole lineage and its size follows the depth of the lineage, not the mission time.

The reader (rrestf.f) rewinds the image, reads the header, and scans the dumps in order until it finds the one whose ncount equals the number on card 103. The first dump it meets must be a complete restart; otherwise it calls fabend and RELAP5 aborts.

All branches of one branch point restore from the same record. Card 103 is built from the restart count stored in the branch cluster (BranchCluster::getNCountForRestartRELAP5()), and the branches differ in the alternative they take, not in the state they resume from.

Why it applies to MPI only

The serial engine captures one image per sequence, after the sequence ends (ADSIDAC.cpp), and that image serves every branch point of the sequence. Each branch point restores from a different record of it, so the serial engine needs the whole chain.

An MPI worker captures an image at each branch point (AdsidacCallback::onBranchingPoint) and the image is used for the branches of that branch point only, which all restore from one record. Everything else in the image is unused by the receiver. The trimming code sits inside the MPI branch of onBranchingPoint (worker ranks only), so a serial run ignores ADS_RESTART_TRIM.

What is shipped

With ADS_RESTART_TRIM=1 (read once per process), the donor worker does the following at every branch point, before the image is stored or sent:

1. It looks up the ncount of the branch cluster.
2. adsidac::extract_record_no_base() (src/ADSIDAC/Global/RestartTrim.h) walks the record chain and returns [header][target record]: the header block followed by the one dump whose ncount matches. A dump extends from its 8-byte header record to the start of the next dump, or to the end of the image.
3. The result replaces the captured image.

Because the replacement happens before the engine chooses between memory and file storage, the smaller payload is what the worker holds in held_data_, what it writes to a restart_*.r transfer file in file mode, what it sends to the peer, and what a DET checkpoint stores for a pending branch. With --compress-queue the LZ4 compression is applied to the trimmed payload.

On the BV4 three-loop PWR deck a trimmed payload is 2.59 MB (0.25 MB header plus one 2.34 MB dump), whatever the depth of the lineage.

Fallback. If no dump with the requested ncount is found, the image is left untouched and the worker sends it whole. The rank log then contains a line that starts with [TRIM] cid=<cluster id> ncount=<n> not found in buffer. The run stays correct; only that transfer is larger.

The run-wide base

[header][first complete record] is the same in every image of a run, because every lineage descends from the same first sequence. The engine relies on this: the base is kept off the wire and distributed once through the project directory.

• The first time a worker process reaches a branch point with trimming on, it checks for <project-dir>/restart_base.bin. If the file does not exist, the worker extracts the base from its own image (adsidac::extract_header_and_base()), writes it to restart_base.bin.r<rank>.tmp and renames it to restart_base.bin. Several ranks can reach this point at the same time; the temporary name is per rank and the rename is atomic, so readers never see a partial file. If the rename fails the temporary file is removed.
• An existing restart_base.bin is never overwritten, and the engine never deletes the file.
• The file is written before the first transfer that depends on it, so a peer on another node finds it as long as the project directory is on a filesystem shared by all nodes.
• Each worker keeps the base in memory (Client::base_cache_). The cache is filled from the first self-contained image the worker restores, or from restart_base.bin when a payload without a base arrives first (Client::ensure_base_cache()).

On the BV4 deck the file is about 5.1 MB.

Restoring a trimmed payload

Client::restore_buffer() inspects every image before it reaches the solver. adsidac::has_complete_restart() reports whether the image contains a complete restart record.

• If it does, the image is self-contained (an untrimmed image, or a payload whose target record is itself a complete restart). The worker remembers its base when the cache is still empty and restores the image unchanged.
• If it does not, adsidac::splice_base() builds [header][complete record][target record]: the cached base followed by the payload without its own copy of the header. This satisfies the rule in rrestf.f that the first dump read is a complete restart, and the scan then reaches the target record and restores from it.

The restored sequence appends its own dumps to the spliced image, so with trimming on a working image holds the base, one inherited record and the records of the current sequence.

If a payload without a base arrives and neither the cache nor restart_base.bin can supply one, the worker prints

FATAL: rank <n> received a base-less restart buffer and has no cached base (<path> missing)

and calls MPI_Abort. Restoring such an image would abort inside RELAP5.

RelapSolver::restore() applies the same check and the same splice, reading restart_base.bin on demand, so a restore is protected on every path into the solver, including those that do not pass through Client. It calls std::abort() when no base is available.

Consequences for users

• Delete restart_base.bin when the input deck or the plant model changes. The engine creates the file only when it is absent and does not compare it with the current deck, and the file is not part of the configuration fingerprint. A base left over from a different model would be spliced onto the new run's payloads.
• Keep restart_base.bin with a checkpoint that will be resumed. The checkpoint stores trimmed payloads for the pending branches, and the resume needs the base to complete them.
• The project directory must be visible to every rank.
• Environment variables are read once per process. Under mpirun across nodes, forward the variable to the ranks (-x ADS_RESTART_TRIM with Open MPI).

Diagnostic variables

Variable	Effect
ADS_MEMBUF_TRACE=1	Reports what a restart image is made of. At every capture RelapSolver::checkpoint() prints [MEMBUF] capture <MB> dumps=<n> rstrec=<MB> rnewp=<MB> wrplid=<MB> (per dump <MB>) on standard output: the size of the captured image, and the dumps and bytes that the three RELAP5 writers (rstrec, rnewp, wrplid) have added since the buffer was last reset or restored. With trimming on, every trimmed transfer adds [TRIM] cid=<id> ncount=<n> <MB> -> <MB> (<ratio>x). The Fortran module writes its own [MEMBUF] lines (reset, capture, restore, start offset of each dump) to Fortran unit 6, which RELAP5 rebinds to its own output, so the C++ lines are the ones to read.
ADS_MEMBUF_TRIM=1	Experimental, off by default. Makes every capture, serial or MPI, return [header][last dump] through ads_membuf_get_trimmed_ (the whole image when it holds fewer than two dumps). The result is valid only when the image is restored at the record it was captured at, so it does not suit the serial engine, which restores the branch points of one sequence from different records of one image. Do not use it for tree generation.

Set them in the environment before the run starts. They are independent of ADS_RESTART_TRIM.

Testing

No ctest case sets ADS_RESTART_TRIM; the validation suite in tests/CMakeLists.txt runs with trimming off. The scripts in tests/scripts/ (validate_mpi.sh, validate_checkpoint.sh, validate_merge.sh, validate_pe_aggregate.sh) start mpirun with the caller's environment, so the MPI cases run with trimming on when the variable is exported first:

cd build
ADS_RESTART_TRIM=1 ctest -R "BV4_MPI" -V

The pass criteria are unchanged, because trimming changes what is transported and not the tree: the same ZESNode, ZPENode, KPS and ES_summary.txt counts are expected. The scripts do not remove restart_base.bin from the project directory they run in (build/tests/BV4/ for most cases), so delete it there when the deck changes.