Skip to main content

Module plan

Module plan 

Source
Expand description

Config-driven load planning — derive a BigQuery load (native schema, table, partition, source URIs) from a rivet export config, so a client never hand-types column types. The schema comes from rivet’s own type resolver via rivet check --target bigquery --json (the argv/process boundary, ADR-0026); the table/partition/destination come from the parsed config.

Structs§

LoadPlan
What a rivet config resolves to for a BigQuery load.
LoadSection
The warehouse load target — the config’s top-level load: block, declared ONCE for all exports. OSS accepts and ignores this block (a reserved passthrough); the loader reads it here. One config file drives both the export and the load — no second file, no per-table repetition.

Enums§

LoadMode
Which load strategy an export’s mode maps to. Drives BOTH the ledger’s file selection and the warehouse write path:
LoadTarget
A warehouse and its connection config. target: is the serde discriminator.

Functions§

plan_loads
source_engine
Resolve the config’s source engine into the CDC [SourceEngine] the dedup view’s __pos parse is keyed on. One config has one source, so this is a job-wide property. MongoDB is supported too: its change stream carries a document _id (the dedup partition key) and an order-preserving _data resume token in __pos, so the current-state view applies just as it does to the relational engines.