Troubleshooting
Each entry gives the message as the engine prints it, the cause, and the remedy. Runtime model errors are also appended to err messages.txt (in the launch directory for serial runs, in Output/ES/ after an MPI merge).
Startup
| Message or symptom | Cause | Remedy |
|---|---|---|
FATAL: input deck path is <n> characters, exceeding RELAP5's 80-character filename limit | RELAP5 stores file names in 80-character buffers. The limit applies to the absolute path, because --input-deck is made absolute | Move the project to a shorter path |
FATAL: restart path exceeds RELAP5's 80-character filename limit | Same limit, for the in-memory restart file. Its name starts with /dev/shm/adsidac_<pid>_ where /dev/shm exists and with the system temporary directory elsewhere (macOS) | Set a shorter TMPDIR, or use --no-memory-restart |
FATAL: input deck not found or unreadable: <path> | Wrong --input-deck | Check the path; for BV4 it is tests/BV4/BV4.i |
FATAL: malformed ZCtrlPar.txt | Input/Scheduler/ZCtrlPar.txt does not contain a parsable Truncation_probability and Truncation_time | Restore the three-line format used by tests/BV4 |
| RELAP5 stops immediately, steam table file not found | The working directory has no tpfh2o, tpfh2onew, tpfd2o | Launch from a directory that holds them, or copy them into the project directory for MPI runs. See Working directory and steam tables |
| macOS: the binary fails at load with a missing Boost or MPI symbol | Homebrew upgraded boost, gcc or open-mpi after the build | Reconfigure in place (cmake . -U "MPI*" -DCMAKE_Fortran_COMPILER=/opt/homebrew/bin/gfortran-NN), rebuild, and rerun ctest -R BV4_Serial_Validate |
Startup never finishes and prints getSupportSystemReliabilityUnit NONE can not be found without end | Initiating_Event.txt ends right after the last event, without the two trailer tokens | Add the trailer described in Initiating_Event.txt |
WARNING: SystemReliability.txt: malformed or unexpected content where <reader> expected a section header | A section of SystemReliability.txt is missing or out of order; the failure models from that point on are not loaded | Compare the section order with SystemReliability.txt. The run continues without those models |
During a run
| Message or symptom | Cause | Remedy |
|---|---|---|
Every sequence ends near 0.5 s with Operator::checkCompletenessOfSystemDecomposition err. and consequence -1; err messages.txt lists The following ControlPanel elements do not exist ... | A control panel element (indicator, control, alarm, support-system state) has no entry in an operator's system decomposition | Add each listed name to both KB_ODM_System_Decomposition.txt and KB_OAT_System_Decomposition.txt |
<name> can not be found! | An alarm, display or knowledge-base entry refers to a control panel name that does not exist | Correct the name in the input file that refers to it. adsmd validate checks most cross-references between input files before a run |
ERROR: write failed (disk full or I/O error): <path>, with free space on the disk | A file-count quota is exhausted | Check the group quota, free files, and use --aggregate-pe. See File count and capacity |
FATAL: failed to write <n> bytes to /dev/shm/adsidac_<pid>_... (disk full?) | On Linux the in-memory restart files live in /dev/shm, and it is full. Containers give it 64 MB by default, less than two restart images | Start the container with a larger segment (docker run --shm-size=2g), or run with --no-memory-restart |
FATAL: rank <n> received a base-less restart buffer and has no cached base (<project>/restart_base.bin missing) | ADS_RESTART_TRIM=1 and restart_base.bin was deleted during the run or is not visible on that node | Keep the project directory on a filesystem shared by all nodes, and do not remove restart_base.bin while a run or a checkpoint chain is in progress |
Wrong restarts after changing the input deck with ADS_RESTART_TRIM=1 | A restart_base.bin from the earlier deck is still in the project directory; the engine creates the file only if it does not exist | Delete restart_base.bin whenever the deck or plant model changes |
[TRIM] cid=<n> ncount=<m> not found in buffer followed by sending it whole | The record to ship was not located; the worker falls back to the full restart image | None needed; the run is correct, that transfer is larger |
WARNING: cannot open occupancy log <path> | The path is not writable from rank_0/ | Pass an absolute path to --occupancy-log |
Input file name is not = indeck.i once per sequence | RELAP5 notes that the deck name differs from its default | None; the line is informational |
| MPI run exits with code 138 when the batch system signals it | mpirun received SIGUSR1 and exited | End slices with --checkpoint-interval and --checkpoint-stop. See Chained slices |
Merge and outputs
| Message or symptom | Cause | Remedy |
|---|---|---|
ERROR: --merge-only runs serially; launch it without mpirun | --merge-only was started under MPI | Run it as a plain process |
ERROR: no rank_N directories under <project>; nothing to merge | There is nothing to consolidate and Output/ is not a finished run | Check --project-dir |
[MERGE-ONLY] Output/ is already finalized ...; nothing to do | The merge has already completed | None |
ERROR: Output/PE already carries a finalized run while rank directories still hold outputs | The rank directories belong to a different run than Output/ | Move Output/ aside, then run --merge-only again |
[MERGE-ONLY] no MERGE_PENDING.txt manifest: salvage mode | The run ended without writing the manifest (for example it was killed) | None; pass --aggregate-pe if the run used it |
[MERGE] failed to move <path>: <reason> | A rename or copy failed, typically on a quota | Free files or space and run --merge-only |
find -name 'ZPENode*' | wc -l reports 1 | With --aggregate-pe the nodes are records in ZPENodes_agg.txt, which the pattern also matches | python3 tests/scripts/pe_canonical.py Output/PE | wc -l |
Serial and MPI ZESNode_<n>.txt differ | Sequence ids are assigned in a different order | Compare by content. See How MPI output compares to serial |
Checkpoint and resume
| Message or symptom | Cause | Remedy |
|---|---|---|
FATAL: checkpoint configuration mismatch | The input deck, ZCtrlPar.txt or a file under Input/ changed since the checkpoint | Restore the inputs, or start a fresh run |
FATAL: the project Output/ already contains a finalized run | The checkpoint's run has completed in this Output/ | Move Output/ aside to rerun from the checkpoint |
FATAL: checkpoint version <v> ... cannot be read by this binary | The file was written in an older format; the loader accepts version 9 only | Finish that run with the binary that wrote it, or start a fresh run |
FATAL: checkpoint file is corrupt or truncated | The job was killed while the file was being written, or the file was copied incompletely | Resume from the previous checkpoint |
ERROR: this project has a deferred, unmerged run (Output/MERGE_PENDING.txt) | A --merge=defer run has not been consolidated | Run --merge-only first |
[RESUME] Overriding --compress-queue to match the checkpoint | The resume command and the checkpoint disagree on --compress-queue | None; the checkpoint's setting is used |
| Resumed sequences end at a different time from the earlier ones | --truncation-time was not repeated on the resume command | Rerun the resume with the same value |
No checkpoint_seq_*.dat after a slice hit its wall limit | The limit was reached while the server waited for the workers to go idle | Lower --checkpoint-interval so the slice ends with margin of one longest sequence |
| A serial resume of an MPI checkpoint never finishes and keeps writing sequences that end with "terminated for unknown reason" | Resuming an MPI checkpoint in serial mode is not supported | Stop it and resume under MPI; mpirun -n 2 gives one worker |