Functions
const char *	manifest_basename ()
	File name of the sentinel written to Output/ by –merge=defer. Its presence means the rank directories are the only complete copy of the run's outputs; Scheduler::parseOptions refuses –resume while it exists. More...

std::vector< std::string >	append_basenames ()
	Base names of the outputs that are merged by concatenation rather than by moving: the three analysis logs (under Output/ES) and the PE aggregate (under Output/PE). More...

bool	is_append_basename (const std::string &fname)
	True if `fname` (a base name, no directory) is one of append_basenames(). More...

std::vector< int >	discover_ranks (const std::string &project_dir)
	Worker rank numbers N for which `<project_dir>/rank_N` is a directory. More...

void	move_file (const fs::path &src, const fs::path &dest, std::unordered_set< std::string > &made_dirs)
	Move one file: rename (replacing an existing `dest`), with copy+remove fallback for EXDEV or exotic filesystems. More...

void	move_rank_es_kps (const std::string &project_dir, int r)
	Move one rank's ES and KPS files into the shared tree, keeping their paths relative to Output/ES and Output/KPS. More...

void	move_rank_pe_files (const std::string &project_dir, const std::vector< int > &ranks)
	Rank-ordered move of per-file PE nodes into Output/PE. More...

void	concat_appends_from_ranks (const std::string &project_dir, const std::vector< int > &ranks)
	Concatenate every rank's append files into the shared Output/. More...

size_t	build_es_summary (const std::string &dest_es)
	Rebuild `<dest_es>ES_summary.txt` by scanning the destination ES tree. More...

void	write_manifest (const std::string &project_dir, int ranks, bool aggregate, long sequences, unsigned long long fingerprint)
	Write the –merge=defer manifest `<project_dir>/Output/MERGE_PENDING.txt` (Output/ is created if needed; an existing manifest is overwritten). More...

bool	manifest_exists (const std::string &project_dir)
	True if `<project_dir>/Output/MERGE_PENDING.txt` exists. More...

bool	manifest_aggregate (const std::string &project_dir, bool fallback)
	The aggregate_pe flag the deferred run recorded in its manifest. More...

bool	looks_finalized (const std::string &pe_dir)
	True when `pe_dir` already carries the finalized signature: at least one node, ids dense 0..N-1, and no negative (-999 sentinel) id. More...

bool	ranks_hold_outputs (const std::string &project_dir, const std::vector< int > &ranks)
	True when any `rank_N/Output` tree still holds mergeable output files (names starting with ZESNode, ZKPS, ZDIA, ZPIF or ZPENode, or an append source). After a successful merge the rank dirs remain (logs, fort files) but hold none of these. More...

std::pair< size_t, size_t >	merge_project (const std::string &project_dir, bool aggregate, int threads)
	The full standalone merge, in this order: ES/KPS moves (threaded over rank dirs), rank-ordered PE moves, append concatenation, ES_summary rebuild, PE finalize (sequential renumber), manifest removal. More...

Function Documentation

◆ append_basenames()

std::vector< std::string > adsidac::merge::append_basenames ( )

inline

Base names of the outputs that are merged by concatenation rather than by moving: the three analysis logs (under Output/ES) and the PE aggregate (under Output/PE).

Definition at line 60 of file OutputMerge.h.

References adsidac::pe::aggregate_basename().

Referenced by concat_appends_from_ranks(), and is_append_basename().

Here is the call graph for this function:

Here is the caller graph for this function:

◆ build_es_summary()

size_t adsidac::merge::build_es_summary ( const std::string & dest_es )

inline

Rebuild <dest_es>ES_summary.txt by scanning the destination ES tree.

Shared by Server::merge_outputs and merge_project so the in-process and standalone paths produce byte-identical summaries. Overwrites the file with a tab-separated header line ("ES#, Probability, End_time, Consequence#, Description") followed by one line per ZESNode_<id>.txt file found recursively under dest_es, in ascending id order. A ZESNode file with fewer than two lines is counted but contributes no summary line.

Parameters

dest_es ES directory WITH a trailing '/' (the file name is appended to the string as is).

Returns: Number of ZESNode files found (the sequence count).

Definition at line 265 of file OutputMerge.h.

Referenced by Server::merge_outputs(), and merge_project().

Here is the caller graph for this function:

◆ concat_appends_from_ranks()

void adsidac::merge::concat_appends_from_ranks	(	const std::string &	project_dir,
		const std::vector< int > &	ranks
	)

inline

Concatenate every rank's append files into the shared Output/.

For each analysis log that has at least one source, the destination is TRUNCATED and each rank's source is appended in the order of ranks; the PE aggregate destination is opened in append mode (it accumulates across checkpoint phases) and each rank's aggregate is compacted first (pe::compact_aggregate; the total is logged on stdout when it shrank). A rank's source is its rank dir copy or, if that is absent, the part a –merge=now worker staged as Output/<ES|PE>/parts/<base>.rank_N. Each source is deleted once streamed, and an emptied parts/ directory is removed. Finally every rank's "err messages.txt" (rank cwd, server rank_0 included, and staged parts) is appended to Output/ES/err messages.txt and deleted; that destination is opened, and so created, even when no rank has one.

Definition at line 184 of file OutputMerge.h.

References adsidac::pe::aggregate_basename(), append_basenames(), and adsidac::pe::compact_aggregate().

Referenced by merge_project().

Here is the call graph for this function:

Here is the caller graph for this function:

◆ discover_ranks()

std::vector< int > adsidac::merge::discover_ranks ( const std::string & project_dir )

inline

Worker rank numbers N for which <project_dir>/rank_N is a directory.

Returns: Ascending list with N > 0 only (rank_0 is the server and holds no outputs). Empty if the project directory cannot be read or has none.

Definition at line 75 of file OutputMerge.h.

References e.

Referenced by merge_project(), and run_merge_only().

Here is the caller graph for this function:

◆ is_append_basename()

bool adsidac::merge::is_append_basename ( const std::string & fname )

inline

True if fname (a base name, no directory) is one of append_basenames().

Definition at line 66 of file OutputMerge.h.

References append_basenames().

Referenced by move_rank_es_kps(), move_rank_pe_files(), and ranks_hold_outputs().

Here is the call graph for this function:

Here is the caller graph for this function:

◆ looks_finalized()

bool adsidac::merge::looks_finalized ( const std::string & pe_dir )

inline

True when pe_dir already carries the finalized signature: at least one node, ids dense 0..N-1, and no negative (-999 sentinel) id.

Reads ids from ZPENode_<id>.txt file names and from the records of a PE aggregate file, so it works in either layout. Same test as the resume guards in ADSIDAC.cpp and Server::dispatch_resume_tasks. A missing directory returns false.

Definition at line 368 of file OutputMerge.h.

References adsidac::pe::aggregate_basename(), and adsidac::pe::parse_aggregate().

Referenced by run_merge_only().

Here is the call graph for this function:

Here is the caller graph for this function:

◆ manifest_aggregate()

bool adsidac::merge::manifest_aggregate	(	const std::string &	project_dir,
		bool	fallback
	)

inline

The aggregate_pe flag the deferred run recorded in its manifest.

Returns: The recorded flag, or fallback when there is no manifest (salvage of an interrupted merge) or the line is missing. run_merge_only passes the –aggregate-pe value of the merge-only command line as the fallback.

Definition at line 350 of file OutputMerge.h.

References manifest_basename().

Referenced by run_merge_only().

Here is the call graph for this function:

Here is the caller graph for this function:

◆ manifest_basename()

const char * adsidac::merge::manifest_basename ( )

inline

File name of the sentinel written to Output/ by –merge=defer. Its presence means the rank directories are the only complete copy of the run's outputs; Scheduler::parseOptions refuses –resume while it exists.

Definition at line 55 of file OutputMerge.h.

Referenced by manifest_aggregate(), manifest_exists(), merge_project(), run_merge_only(), and write_manifest().

Here is the caller graph for this function:

◆ manifest_exists()

bool adsidac::merge::manifest_exists ( const std::string & project_dir )

inline

True if <project_dir>/Output/MERGE_PENDING.txt exists.

Definition at line 341 of file OutputMerge.h.

References manifest_basename().

Referenced by run_merge_only().

Here is the call graph for this function:

Here is the caller graph for this function:

◆ merge_project()

std::pair< size_t, size_t > adsidac::merge::merge_project	(	const std::string &	project_dir,
		bool	aggregate,
		int	threads
	)

inline

The full standalone merge, in this order: ES/KPS moves (threaded over rank dirs), rank-ordered PE moves, append concatenation, ES_summary rebuild, PE finalize (sequential renumber), manifest removal.

Every step consumes its sources, so rerunning after an interruption picks up what is left. One limit: concat_appends_from_ranks truncates an analysis log whenever a source for it remains, so blocks an interrupted pass had already concatenated (and deleted at the source) are not kept by the rerun. The caller (run_merge_only in ADSIDAC.cpp) first checks that Output/PE is not already finalized; this function does not.

Parameters

project_dir	Project directory holding Output/ and the rank_N dirs.
aggregate	PE layout to write: true = one aggregate file, false = one file per node.
threads	Worker threads for the ES/KPS moves; values below 1 mean 1, and no more threads than rank dirs are started. The remaining steps are single-threaded.

Returns: (sequence count from build_es_summary, PE node count from pe::finalize_sequential).

Definition at line 440 of file OutputMerge.h.

References build_es_summary(), concat_appends_from_ranks(), discover_ranks(), adsidac::pe::finalize_sequential(), manifest_basename(), move_rank_es_kps(), and move_rank_pe_files().

Referenced by run_merge_only().

Here is the call graph for this function:

Here is the caller graph for this function:

◆ move_file()

void adsidac::merge::move_file	(	const fs::path &	src,
		const fs::path &	dest,
		std::unordered_set< std::string > &	made_dirs
	)

inline

Move one file: rename (replacing an existing dest), with copy+remove fallback for EXDEV or exotic filesystems.

A failure is reported on stderr ("[MERGE] failed to move ...") and leaves src in place; it does not abort the merge.

Parameters

src	File to move.
dest	Destination path; its parent directory is created if needed.
made_dirs	Caller-owned cache of parent directories already created, so each is created once. Not shared between threads.

Definition at line 100 of file OutputMerge.h.

Referenced by move_rank_es_kps(), move_rank_pe_files(), and Client::self_merge_outputs().

Here is the caller graph for this function:

◆ move_rank_es_kps()

void adsidac::merge::move_rank_es_kps	(	const std::string &	project_dir,
		int	r
	)

inline

Move one rank's ES and KPS files into the shared tree, keeping their paths relative to Output/ES and Output/KPS.

File names carry the global sequence id, so ranks never collide and calls for different ranks may run concurrently (merge_project does). Append files and everything under PE are left alone: appends are concatenated in rank order, and per-file PE winner selection must stay rank-ordered. Does nothing if rank_<r>/Output does not exist.

Definition at line 125 of file OutputMerge.h.

References is_append_basename(), and move_file().

Referenced by merge_project().

Here is the call graph for this function:

Here is the caller graph for this function:

◆ move_rank_pe_files()

void adsidac::merge::move_rank_pe_files	(	const std::string &	project_dir,
		const std::vector< int > &	ranks
	)

inline

Rank-ordered move of per-file PE nodes into Output/PE.

The same filename exists on several ranks in different completeness states (clients re-save whole lineages), and the deterministic winner is the highest rank: rename-over-existing in the order of ranks gives the same result as the –merge=serial copy loop. Pass ranks ascending and call from one thread. The PE aggregate file is skipped (it is an append file).

Definition at line 152 of file OutputMerge.h.

References is_append_basename(), and move_file().

Referenced by merge_project().

Here is the call graph for this function:

Here is the caller graph for this function:

◆ ranks_hold_outputs()

bool adsidac::merge::ranks_hold_outputs	(	const std::string &	project_dir,
		const std::vector< int > &	ranks
	)

inline

True when any rank_N/Output tree still holds mergeable output files (names starting with ZESNode, ZKPS, ZDIA, ZPIF or ZPENode, or an append source). After a successful merge the rank dirs remain (logs, fort files) but hold none of these.

Definition at line 404 of file OutputMerge.h.

References is_append_basename().

Referenced by run_merge_only().

Here is the call graph for this function:

Here is the caller graph for this function:

◆ write_manifest()

void adsidac::merge::write_manifest	(	const std::string &	project_dir,
		int	ranks,
		bool	aggregate,
		long	sequences,
		unsigned long long	fingerprint
	)

inline

Write the –merge=defer manifest <project_dir>/Output/MERGE_PENDING.txt (Output/ is created if needed; an existing manifest is overwritten).

Text format, one "key value" pair per line after a "# adsidac deferred merge v1" comment: ranks, aggregate_pe (0/1), sequences, config_fingerprint (hex). Only aggregate_pe is read back (manifest_aggregate); the other fields are informational.

Parameters

project_dir	Project directory that contains (or will contain) Output/.
ranks	Rank count to record. parallelexec passes the MPI world size, which includes the server rank.
aggregate	Whether the run wrote PE nodes in the aggregate layout.
sequences	Sequences completed by the run.
fingerprint	Configuration fingerprint of the run (g_sim.config_fingerprint).

Definition at line 327 of file OutputMerge.h.

References manifest_basename().

Here is the call graph for this function:

Functions

Function Documentation

◆ append_basenames()

◆ build_es_summary()

◆ concat_appends_from_ranks()

◆ discover_ranks()

◆ is_append_basename()

◆ looks_finalized()

◆ manifest_aggregate()

◆ manifest_basename()

◆ manifest_exists()

◆ merge_project()

◆ move_file()

◆ move_rank_es_kps()

◆ move_rank_pe_files()

◆ ranks_hold_outputs()

◆ write_manifest()