Crate knut_bifrost 

knut-bifrost

The bridge between the table world and the graph world: watch lakehouse tables (Iceberg via Spark Connect) and continuously materialise them as a FalkorDB property graph, through a declarative-or-custom transform fabric.

See docs/bifrost-design.md for the full design.

use std::time::Duration;
use knut_bifrost::{
    Listener, MappingSpec, TransformRegistry, SyncState,
    sink::FalkorDbSink, source::SparkConnectSource,
};

let spec = MappingSpec::load("bifrost.yml")?;
let source = SparkConnectSource::connect("sc://spark:15002").await?;
let sink = FalkorDbSink::connect("redis://127.0.0.1:6379", "social").await?;
let state = SyncState::open("bifrost-state.json")?;

let mut listener = Listener::new(spec, TransformRegistry::new(), source, sink, state);
listener.run(Duration::from_secs(30), |round| {
    println!("{round:?}");
    true
}).await?;

Re-exports

pub use airflow::AssetEmitter;

pub use airflow::AssetEvent;

pub use airflow::emit_round;

pub use airflow::round_to_events;

pub use cdc::CdcBackend;

pub use cdc::CdcCapability;

pub use cdc::CdcKind;

pub use cdc::ChangeOp;

pub use cdc::ChangeRequest;

pub use cdc::ChangeStream;

pub use cdc::change_row_schema;

pub use cdc::encode_change_rows;

pub use cdc::fold_updates;

pub use codegen::CodegenOptions;

pub use codegen::EdgePlan;

pub use codegen::EdgeProposal;

pub use codegen::GraphPlan;

pub use codegen::NodePlan;

pub use codegen::PlanMode;

pub use codegen::SinkKind;

pub use codegen::SourceKind;

pub use codegen::Tier;

pub use error::BifrostError;

pub use error::Result;

pub use listener::Listener;

pub use listener::RoundReport;

pub use source::ArrowBatches;

pub use source::ArrowSource;

pub use source::EmptyRead;

pub use source::TableSpec;

pub use mapping::MappingSpec;

pub use mapping::PipelineStep;

pub use mapping::SyncMode;

pub use mapping::TableMapping;

pub use metrics::Backoff;

pub use metrics::Metrics;

pub use metrics::render_prometheus;

pub use mutation::EdgeSet;

pub use mutation::GraphMutation;

pub use mutation::SinkReport;

pub use mutation::VertexDelete;

pub use mutation::VertexSet;

pub use registry::TransformRegistry;

pub use rel2graph::ArrowTables;

pub use rel2graph::BuildReport;

pub use rel2graph::TableSource;

pub use rel2graph::build_mutation;

pub use rel2graph::build_mutation_reported;

pub use source::FileSource;

pub use state::SyncState;

pub use state::TableState;

pub use transform::ChangeRows;

pub use transform::EdgeTransform;

pub use transform::TableContext;

pub use transform::Transform;

pub use transform::VertexTransform;

pub use transform::deletes_from_change_rows;

pub use transform::edge_deletes_from_change_rows;

pub use transform::is_changelog_batch;

pub use transform::partition_change_rows;

Modules

airflow

Airflow 3 asset events (AIP-82) — the emit half of the story.

cdc

The CDC seam — one change-data-capture vocabulary, three backends, and the Arrow row shape a UI renders change-by-change.

codegen

rel2graph CODEGEN — one MappingSpec projected onto executable backends.

error

listener

The sync loop: poll → transform → sink → advance state.

mapping

Declarative YAML mapping spec — the fabric’s blueprint.

metrics

Cumulative run metrics and the poll-loop’s failure backoff schedule.

mutation

The common currency between transforms and sinks.

puppygraph

PuppyGraph schema.json as a bifröst mapping.

registry

Named registry for custom Rust transforms referenced from YAML mappings.

rel2graph

The canonical offline rel2graph engine seam.

sink

Sinks: the two places a bifröst round can land.

source

Table sources. SparkConnectSource (feature spark) polls Iceberg tables through a Spark Connect 4.x endpoint.

spark_demo

The knut ▸ Spark 4.1 + Iceberg 1.11.0 LOAD demo — one shared source of truth.

state

Per-table sync progress — “what is already done”.

testmatrix

Functional-status emitters for the nornir-testmatrix matrix.

transform

The transform fabric: built-in declarative transforms plus the trait custom Rust transforms implement.

Constants

GIT_HASH

The short git commit this crate was built from (KNUT_GIT_HASH, emitted by build.rs via a pure-fs, zero-shell read of the repo’s git metadata). env! (not option_env!) because build.rs emits it on every build.

VERSION_LINE

--version line: v<pkg-version> (<git-sha>). clap prints <bin> <this>, so --version reads <bin> v<ver> (<hash>) — the suite-wide identity.