fv-connect 0.1.0

FusionVault Conduit: read relational, file and REST sources into rows with mandatory access control pushed into the read, write results back with atomic promotion, and drive incremental base-ingest scans, changelogs and label normalisation checked against the published conformance vectors.
Documentation

fv-connect · FusionVault Conduit

fv-connect is how FusionVault reads and writes the systems its data lives in. It connects to a relational database, a SQLite file or a REST endpoint, reads rows with the caller's access-control filter pushed into the query so hidden rows never leave the source, writes transform outputs back with an atomic swap so readers never see a half-written table, and drives the incremental base-ingest scan that turns successive snapshots into a changelog.

Read a source with access control pushed down

use fv_connect::{read_source, ConnectorConfig};
use fv_mac::{Bindings, Taxonomy};
# async fn demo(taxonomy: Taxonomy, grants: Bindings) -> Result<(), fv_connect::ConnectError> {

let config = ConnectorConfig {
    kind: "POSTGRES".into(),
    host: "localhost".into(),
    port: 5432,
    database: "shop".into(),
    user: "app".into(),
    options: serde_json::Value::Null,
};

// The taxonomy and the subject's grants become a WHERE clause the database evaluates.
let rows = read_source(&config, "app-secret", "curated.orders", &taxonomy, &grants,
                       |axis| format!("{axis}_level"), Some(1000)).await?;
for row in &rows {
    println!("{:?}", fv_connect::get(row, "orderId"));
}
# Ok(()) }

A row is Vec<(String, Value)> in the value dialect, so it flows straight into pipelines. The secret never lives in the config: secret_env_var(name, key) names the environment variable a vault stand-in resolves it from.

Write back with atomic promotion

use fv_connect::{write_and_promote, ConnectorConfig, Row};
# async fn demo(config: ConnectorConfig, rows: Vec<Row>) -> Result<(), fv_connect::ConnectError> {

let columns = vec!["customerId".to_string(), "tier".to_string()];
write_and_promote(&config, "app-secret", "curated.customer_tiers", &columns, &rows).await?;
# Ok(()) }

Rows land in a staging table first; the swap into place is a single transaction, so the previous good output stays current until the new one is complete.

Sources are a registry

Each connector type is a Source: what it can do (Caps), which scan leg drives it, and how it reads and writes. Postgres, MySQL, SQLite and REST are registered at startup; a new one is one register_source call, and every entry point dispatches through the registry.

use std::sync::Arc;
use fv_connect::{register_source, registered_kinds, Caps, ConnectorSource};
# struct MySource;
# #[async_trait::async_trait]
# impl ConnectorSource for MySource {
#     fn capabilities(&self) -> Caps { Caps { read: true, write: false } }
#     async fn read(&self, _: &fv_connect::ConnectorConfig, _: &str, _: &str, _: &fv_mac::Taxonomy, _: &fv_mac::Bindings, _: fv_connect::source::LabelColumn<'_>, _: Option<usize>) -> Result<Vec<fv_connect::Row>, fv_connect::ConnectError> { Ok(vec![]) }
#     async fn write_and_promote(&self, _: &fv_connect::ConnectorConfig, _: &str, _: &str, _: &[String], _: &[fv_connect::Row]) -> Result<(), fv_connect::ConnectError> { Ok(()) }
# }

register_source("MQTT", Arc::new(MySource));
assert!(registered_kinds().contains(&"MQTT".to_string()));

The REST source needs no code at all: its options carry the URL, an optional recordsPath into the response, and a fields map from output column to a value-dialect expression over each record.

Incremental ingest

scan_source reads every snapshot newer than a checkpoint, diffs each against the last with diff_snapshots, and emits property-level INSERT / UPDATE / DELETE changes to a Sink; it returns the new checkpoint. SnapshotStore and Checkpoint are the on-disk landing for polled sources, LandingSource wraps a fetch function into a scannable source, and fold_compacted_log turns a keyed message log into the same snapshot shape. normalize_row applies a datasource binding's label columns and property allow-list to each row.

Bindings, changelog diffs, compacted-log folds and row normalisation are each checked against the published fv-contract conformance vectors, shared with the TypeScript control plane.

Develop

cargo fmt --check && cargo clippy --all-targets -- -D warnings && cargo test
FV_PG_DSN=postgresql://app:app-secret@localhost:5432/shop cargo test --test pg_live -- --ignored

The SQLite and REST paths test in-process. Postgres and MySQL tests are #[ignore] and run against a live database when their DSN variable is set.

Apache-2.0. Plain semver; tags vX.Y.Z at the published commit; see CHANGELOG.md.