Module reconcile 

End-to-end load integrity — reconcile source → file → warehouse row counts before (and after) a load.

The OSS engine records, in each run’s manifest.json, two of the three legs of the chain: how many rows the source held at extraction time (ExtractionMetadata::source_row_count, when cheaply probed) and how many rows were actually written to files (RunManifest::row_count). The warehouse leg — how many rows the load landed — is the loaders’ post-load COUNT(*), enforced by their expected_rows gate.

Until now that gate was dead in the production path: nothing read the manifest, so rivet load trusted whatever Parquet happened to sit under the prefix (“file in a bucket”). This module closes the loop:

1. read every manifest.json under the export’s GCS prefix,
2. refuse to load unless each is a self-consistent Success run whose source count (when known) matches what it extracted, and
3. return the summed, authoritative file_rows the loader’s count gate then checks against the warehouse.

It is the value the OSS core deliberately stops short of — file-level integrity is free (rivet validate); reconciling that the rows arrived in the warehouse is the paid last mile.

Structs

LoadIntegrity

The reconciled row-count chain for one export’s load, derived from the run manifests under its GCS prefix. file_rows is what the warehouse must end up holding; the loader’s expected_rows gate enforces warehouse == file.

Functions

fetch_manifests_keyed

Fetch and parse every manifest.json under gcs_prefix (recursive), keeping each manifest’s bucket-relative storage key — needed to resolve a manifest’s (relative) part paths back to full object keys for per-run loading (see select_load_uris).

gc_orphans

Delete every .parquet under gcs_prefix that no Success manifest references — crash leftovers from an interrupted extract (a run killed before it wrote its manifest leaves orphan parts the load already ignores, but which accumulate). Keeps every manifest, _SUCCESS, and every manifested part (including a snapshot/ sub-prefix’s, since keyed is fetched recursively). Strictly gentler than cleanup_source, which wipes the whole prefix.

latest_full

Full/chunked loads care only about the LATEST snapshot: from keyed (all run manifests under the prefix) pick the newest by finished_at. Full loads OVERWRITE, so exactly ONE snapshot may be loaded (loading every accumulated run would duplicate rows). Deliberately not ledger-gated: full re-materializes the latest snapshot on every load, so a re-load self-heals a drifted target and stays resilient to hidden in-place source updates. An empty input (no staged run — e.g. the staging was cleaned) yields an empty selection, so the caller no-ops WITHOUT truncating the target.

reconcile

Reconcile a run’s manifests into the authoritative expected warehouse row count, refusing to load anything that is not provably complete.

select_load_keys

Pure selection: which bucket-relative parquet keys to load for the given (not-yet-loaded) run manifests.

select_load_uris

Full gs:// URIs of the parquet to load for new (the not-yet-loaded run manifests), preferring each manifest’s own parts over a blanket listing. See select_load_keys for the selection rule.

select_runs

Which run manifests to load for mode, given the ledger’s already-loaded run_ids. The single mode→selection decision, pure and testable so the load’s central invariant isn’t buried in dispatch I/O: