Expand description
Source connectivity — read a relational source into rows with the MAC filter pushed DOWN into the source query, so hidden rows never leave the database.
Local dev connects plaintext (sslmode=disable) via tokio-postgres (NoTls) — no native TLS /
openssl. The MAC WHERE comes from fv_mac::where_sql; for a DOMINANCE axis the fragment
(col <= n) is valid in Postgres as-is. (Array axes ALL_OF/ANY_OF need the Postgres operator
dialect <@ / && — a Postgres-flavoured builder, or a DataFusion custom TableProvider that lets
datafusion-federation translate — is the follow-up; DOMINANCE covers the common curated-table case.)
Re-exports§
pub use binding::bindings_from_definitions;pub use binding::primary_binding;pub use binding::Binding;pub use changelog::diff_snapshots;pub use changelog::Change;pub use changelog::Snapshot;pub use labels::normalize_row;pub use labels::LabelBinding;pub use lake::Checkpoint;pub use lake::SnapshotStore;pub use log::fold_compacted_log;pub use log::LogMessage;pub use scan::scan_source;pub use scan::CollectSink;pub use scan::LandingSource;pub use scan::MemorySource;pub use scan::Sink;pub use scan::Source;pub use source::register_source;pub use source::registered_kinds;pub use source::source_for;pub use source::Caps;pub use source::Source as ConnectorSource;
Modules§
- binding
- Datasource bindings (the object-type schema’s datasource block), pinned to the contract’s
bindingconformance vectors. - changelog
- BASE-CDC changelog diff — the pure core of the base-ingest scan, pinned to the contract’s
changelogconformance vectors. - labels
- Datasource row normalization, pinned to the contract’s
normalize-rowconformance vectors. - lake
- The landing “lake” — snapshot staging + checkpoint for polled sources.
- log
- Compacted-log fold — the pure core of a streaming (Kafka) source.
- rest
- The generic REST connector (Source SPI). Poll an HTTP(S) endpoint, pull an array of records out of the JSON, and map each record to an object row with the value dialect. ONE driver serves both:
- scan
- Incremental base-ingest scan. Drives the changelog: for a source, emit BASE changes for every snapshot newer than the checkpoint, in order, and return the new checkpoint (highest snapshot processed).
- source
- The connector
SourceSPI — one trait + a registry, so adding a driver is “implement the trait + oneregister_sourcecall” instead of growing amatch config.kindin every capability (the read/write dispatches used to be copied across fv-connect and the fv-datafusion worker; they all collapse to registry lookups here). - sqlite
- SQLite source — the “adding a connector is small” DX proof. Embedded (rusqlite
bundled, zero system deps), so it is also the first driver whose integration tests run fully
in-process.
ConnectorConfig.databaseis the database FILE PATH (SQLite has no host/port); a schema-qualified table name likecurated.customeris one literal (quoted) identifier — SQLite has no schemas, and a dot in a quoted name is fine.
Structs§
- Connect
Error - Connector
Config - A connector’s non-secret configuration, as stored in the control plane’s connector registry. Relational drivers use the
host/port/database/user fields; non-relational drivers (e.g. the generic REST connector) carry their
shape-specific settings in
options(url, recordsPath, fields, pollIntervalSec, …).
Functions§
- get
- Convenience: pull the value of one column from a row (Null if absent).
- read_
mysql_ with_ mac - Read
tablefrom MySQL with the MAC filter pushed into the query (DOMINANCEcol<=nis valid MySQL as-is; array axes would need MySQL JSON functions — see the Postgres note). - read_
series - Read a time-series from a PG-wire sidecar plane —
series.observations(equally a Timescale hypertable / PostGIS store; all PG-wire). MAC is pushed into the WHERE via the Postgres-nativefv_mac::pg_where(enforcement point): a plane read can never return an observation the caller couldn’t see. Returns time-ascending rows (ts + value_num/str/geo columns); the caller resolves the primary value. Reuses the same connection plumbing asread_with_mac. - read_
source - Read a registered source with MAC pushed into the query — a registry lookup on the connector
type (Source SPI); a new driver is one
register_sourcecall away, no new match arms. - read_
source_ preferring_ landing - The S12 landing tee — the PIPELINE RUNNER’S dataset read: prefer the funnel’s LANDED snapshot of
table(the exact curated rows the changelog diffed → deterministic inputs, and a rate-limited public API is never double-fetched), falling back to the liveread_sourcewhen the funnel has never landed this table (e.g. a raw relational table outside any object-type binding). NEVER use this from the funnel’s own scan legs — they fetch-then-land, and a landing-preferring read would freeze ingestion on its own previous state (pinned by rest.rs’s regression guard). - read_
with_ mac - Read
table(schema-qualified, e.g.curated.customer) with MAC pushed into the query. - secret_
env_ var - The env var a connector’s secret is resolved from (the vault stand-in):
FV_CONNECTOR_<NAME>_<KEY>upper-cased, non-alphanumerics →_. - write_
and_ promote - Stage + atomically promote
rowsastableon the connector — the write half of the SPI. Replaces the per-capabilitymatch config.kindthe worker carried. - write_
and_ promote_ mysql - Write
rowsinto a MySQLtable(optionally schema-qualified) and atomically promote (backtick idents;RENAME TABLE cur TO old, stg TO curas the atomic swap).columnsfixes output order, else the union of row keys is used. - write_
and_ promote_ postgres - Write
rowsintotable(schema-qualified) on Postgres and atomically promote.columnsfixes the output column order (e.g. the dataset’s declared schema); if empty, the union of row keys is used. A leak-free no-op is a valid outcome (an empty result still swaps in an empty table).
Type Aliases§
- Row
- One source row as ordered (column, value) pairs (matches the engine’s Row shape).