Skip to main content

Crate fv_connect

Crate fv_connect 

Source
Expand description

Source connectivity — read a relational source into rows with the MAC filter pushed DOWN into the source query, so hidden rows never leave the database.

Local dev connects plaintext (sslmode=disable) via tokio-postgres (NoTls) — no native TLS / openssl. The MAC WHERE comes from fv_mac::where_sql; for a DOMINANCE axis the fragment (col <= n) is valid in Postgres as-is. (Array axes ALL_OF/ANY_OF need the Postgres operator dialect <@ / && — a Postgres-flavoured builder, or a DataFusion custom TableProvider that lets datafusion-federation translate — is the follow-up; DOMINANCE covers the common curated-table case.)

Re-exports§

pub use binding::bindings_from_definitions;
pub use binding::primary_binding;
pub use binding::Binding;
pub use changelog::diff_snapshots;
pub use changelog::Change;
pub use changelog::Snapshot;
pub use labels::normalize_row;
pub use labels::LabelBinding;
pub use lake::Checkpoint;
pub use lake::SnapshotStore;
pub use log::fold_compacted_log;
pub use log::LogMessage;
pub use scan::scan_source;
pub use scan::CollectSink;
pub use scan::LandingSource;
pub use scan::MemorySource;
pub use scan::Sink;
pub use scan::Source;
pub use source::register_source;
pub use source::registered_kinds;
pub use source::source_for;
pub use source::Caps;
pub use source::Source as ConnectorSource;

Modules§

binding
Datasource bindings (the object-type schema’s datasource block), pinned to the contract’s binding conformance vectors.
changelog
BASE-CDC changelog diff — the pure core of the base-ingest scan, pinned to the contract’s changelog conformance vectors.
labels
Datasource row normalization, pinned to the contract’s normalize-row conformance vectors.
lake
The landing “lake” — snapshot staging + checkpoint for polled sources.
log
Compacted-log fold — the pure core of a streaming (Kafka) source.
rest
The generic REST connector (Source SPI). Poll an HTTP(S) endpoint, pull an array of records out of the JSON, and map each record to an object row with the value dialect. ONE driver serves both:
scan
Incremental base-ingest scan. Drives the changelog: for a source, emit BASE changes for every snapshot newer than the checkpoint, in order, and return the new checkpoint (highest snapshot processed).
source
The connector Source SPI — one trait + a registry, so adding a driver is “implement the trait + one register_source call” instead of growing a match config.kind in every capability (the read/write dispatches used to be copied across fv-connect and the fv-datafusion worker; they all collapse to registry lookups here).
sqlite
SQLite source — the “adding a connector is small” DX proof. Embedded (rusqlite bundled, zero system deps), so it is also the first driver whose integration tests run fully in-process. ConnectorConfig.database is the database FILE PATH (SQLite has no host/port); a schema-qualified table name like curated.customer is one literal (quoted) identifier — SQLite has no schemas, and a dot in a quoted name is fine.

Structs§

ConnectError
ConnectorConfig
A connector’s non-secret configuration, as stored in the control plane’s connector registry. Relational drivers use the host/port/database/user fields; non-relational drivers (e.g. the generic REST connector) carry their shape-specific settings in options (url, recordsPath, fields, pollIntervalSec, …).

Functions§

get
Convenience: pull the value of one column from a row (Null if absent).
read_mysql_with_mac
Read table from MySQL with the MAC filter pushed into the query (DOMINANCE col<=n is valid MySQL as-is; array axes would need MySQL JSON functions — see the Postgres note).
read_series
Read a time-series from a PG-wire sidecar plane — series.observations (equally a Timescale hypertable / PostGIS store; all PG-wire). MAC is pushed into the WHERE via the Postgres-native fv_mac::pg_where (enforcement point): a plane read can never return an observation the caller couldn’t see. Returns time-ascending rows (ts + value_num/str/geo columns); the caller resolves the primary value. Reuses the same connection plumbing as read_with_mac.
read_source
Read a registered source with MAC pushed into the query — a registry lookup on the connector type (Source SPI); a new driver is one register_source call away, no new match arms.
read_source_preferring_landing
The S12 landing tee — the PIPELINE RUNNER’S dataset read: prefer the funnel’s LANDED snapshot of table (the exact curated rows the changelog diffed → deterministic inputs, and a rate-limited public API is never double-fetched), falling back to the live read_source when the funnel has never landed this table (e.g. a raw relational table outside any object-type binding). NEVER use this from the funnel’s own scan legs — they fetch-then-land, and a landing-preferring read would freeze ingestion on its own previous state (pinned by rest.rs’s regression guard).
read_with_mac
Read table (schema-qualified, e.g. curated.customer) with MAC pushed into the query.
secret_env_var
The env var a connector’s secret is resolved from (the vault stand-in): FV_CONNECTOR_<NAME>_<KEY> upper-cased, non-alphanumerics → _.
write_and_promote
Stage + atomically promote rows as table on the connector — the write half of the SPI. Replaces the per-capability match config.kind the worker carried.
write_and_promote_mysql
Write rows into a MySQL table (optionally schema-qualified) and atomically promote (backtick idents; RENAME TABLE cur TO old, stg TO cur as the atomic swap). columns fixes output order, else the union of row keys is used.
write_and_promote_postgres
Write rows into table (schema-qualified) on Postgres and atomically promote. columns fixes the output column order (e.g. the dataset’s declared schema); if empty, the union of row keys is used. A leak-free no-op is a valid outcome (an empty result still swaps in an empty table).

Type Aliases§

Row
One source row as ordered (column, value) pairs (matches the engine’s Row shape).