Module load 

Warehouse load layer — the TargetLoader seam, its per-warehouse adapters, and the warehouse-neutral load driver.

OSS decides what a column becomes in the warehouse (TargetColumnSpec via ExportTarget::resolve_table). A TargetLoader adapter runs the warehouse-specific load ([bigquery] — free LOAD DATA; [snowflake] — COPY off a GCS external stage). The driver (run_load / run_load_cdc) owns the invariant orchestration — spec validation, the count-integrity gate, the dedup-view wiring, and cleanup ordering — so those invariants are exercised once through a fake adapter, not per warehouse.

Modules

cdc

CDC current-state dedup view.

plan

Config-driven load planning — derive a BigQuery load (native schema, table, partition, source URIs) from a rivet export config, so a client never hand-types column types. The schema comes from rivet’s own type resolver via rivet check --target bigquery --json (the argv/process boundary, ADR-0026); the table/partition/destination come from the parsed config.

reconcile

End-to-end load integrity — reconcile source → file → warehouse row counts before (and after) a load.

Structs

BigQueryLoader

Loads Rivet Parquet into a BigQuery dataset via the bq CLI.

CdcLoadReport

Outcome of a CDC change-log load: rows appended to the <table>__changes log plus the current-state dedup view rebuilt over it.

LoadReport

Outcome of a successful batch load.

SnowflakeLoader

Loads Rivet Parquet from GCS into Snowflake.

Traits

TargetLoader

A warehouse adapter — the small, warehouse-specific seam the driver drives. Dialect + CLI (bq / snow), the external stage, BigQuery’s 4,000-partition batch split, and PARSE_JSON all live behind these primitives.

Functions

build_loader

The one place a resolved plan’s LoadTarget maps to a concrete TargetLoader adapter — wiring partition / cluster / connection / run-id from the config. The count gate and cleanup are the driver’s, so the adapter carries no expected_rows.

open_store

Open the one [GcsStore] a load reuses for reconcile, URI listing, and post-gate cleanup — the single production constructor cli::dispatch calls.

run_load

Batch load driver. Materialize table from uris, gate the landed rows against expected_rows (the reconciled file count; None skips the gate), and — only after the gate passes — clean up the source via cleanup (Some((store, gs_prefix)) to delete, None to keep it).

run_load_cdc

run_load_incremental

Load an INCREMENTAL export’s delta: APPEND the parquet into <table>__changes (reusing the CDC changelog append — the delta’s rows land with NULL __op/ __pos/__seq, which the view drops) and (re)build a current-state view deduped to the latest row per PK by cursor_column. The manifests’ summed row_count gates the appended delta, and cleanup runs (only) after the gate — safe because the ledger, not the file prefix, records what’s loaded.