OutputMerge.h File Reference

Output consolidation for the standalone –merge-only invocation (rank discovery, threaded ES/KPS renames, rank-ordered PE moves, ordered log concatenation, PE finalize) plus two pieces the MPI run shares: the ES_summary builder (Server::merge_outputs) and the deferred-merge manifest (written by parallelexec under –merge=defer). More...

#include <algorithm>
#include <atomic>
#include <filesystem>
#include <fstream>
#include <iostream>
#include <limits>
#include <set>
#include <sstream>
#include <string>
#include <thread>
#include <unordered_set>
#include <utility>
#include <vector>
#include "PEOutput.h"

Include dependency graph for OutputMerge.h:

This graph shows which files directly or indirectly include this file:

Go to the source code of this file.

Namespaces
namespace	adsidac
namespace	adsidac::merge

Functions
const char *	adsidac::merge::manifest_basename ()
	File name of the sentinel written to Output/ by –merge=defer. Its presence means the rank directories are the only complete copy of the run's outputs; Scheduler::parseOptions refuses –resume while it exists. More...
std::vector< std::string >	adsidac::merge::append_basenames ()
	Base names of the outputs that are merged by concatenation rather than by moving: the three analysis logs (under Output/ES) and the PE aggregate (under Output/PE). More...
bool	adsidac::merge::is_append_basename (const std::string &fname)
	True if fname (a base name, no directory) is one of append_basenames(). More...
std::vector< int >	adsidac::merge::discover_ranks (const std::string &project_dir)
	Worker rank numbers N for which <project_dir>/rank_N is a directory. More...
void	adsidac::merge::move_file (const fs::path &src, const fs::path &dest, std::unordered_set< std::string > &made_dirs)
	Move one file: rename (replacing an existing dest), with copy+remove fallback for EXDEV or exotic filesystems. More...
void	adsidac::merge::move_rank_es_kps (const std::string &project_dir, int r)
	Move one rank's ES and KPS files into the shared tree, keeping their paths relative to Output/ES and Output/KPS. More...
void	adsidac::merge::move_rank_pe_files (const std::string &project_dir, const std::vector< int > &ranks)
	Rank-ordered move of per-file PE nodes into Output/PE. More...
void	adsidac::merge::concat_appends_from_ranks (const std::string &project_dir, const std::vector< int > &ranks)
	Concatenate every rank's append files into the shared Output/. More...
size_t	adsidac::merge::build_es_summary (const std::string &dest_es)
	Rebuild <dest_es>ES_summary.txt by scanning the destination ES tree. More...
void	adsidac::merge::write_manifest (const std::string &project_dir, int ranks, bool aggregate, long sequences, unsigned long long fingerprint)
	Write the –merge=defer manifest <project_dir>/Output/MERGE_PENDING.txt (Output/ is created if needed; an existing manifest is overwritten). More...
bool	adsidac::merge::manifest_exists (const std::string &project_dir)
	True if <project_dir>/Output/MERGE_PENDING.txt exists. More...
bool	adsidac::merge::manifest_aggregate (const std::string &project_dir, bool fallback)
	The aggregate_pe flag the deferred run recorded in its manifest. More...
bool	adsidac::merge::looks_finalized (const std::string &pe_dir)
	True when pe_dir already carries the finalized signature: at least one node, ids dense 0..N-1, and no negative (-999 sentinel) id. More...
bool	adsidac::merge::ranks_hold_outputs (const std::string &project_dir, const std::vector< int > &ranks)
	True when any rank_N/Output tree still holds mergeable output files (names starting with ZESNode, ZKPS, ZDIA, ZPIF or ZPENode, or an append source). After a successful merge the rank dirs remain (logs, fort files) but hold none of these. More...
std::pair< size_t, size_t >	adsidac::merge::merge_project (const std::string &project_dir, bool aggregate, int threads)
	The full standalone merge, in this order: ES/KPS moves (threaded over rank dirs), rank-ordered PE moves, append concatenation, ES_summary rebuild, PE finalize (sequential renumber), manifest removal. More...

Detailed Description

Output consolidation for the standalone –merge-only invocation (rank discovery, threaded ES/KPS renames, rank-ordered PE moves, ordered log concatenation, PE finalize) plus two pieces the MPI run shares: the ES_summary builder (Server::merge_outputs) and the deferred-merge manifest (written by parallelexec under –merge=defer).

Definition in file OutputMerge.h.