Expand description
Run a stream pipeline from one TOML file, with no control plane anywhere.
The engine (fv-streams-engine) asks its host four questions through the
ControlPlane trait: what is the pipeline’s topology, which topic backs a dataset, should I
keep running, and here is my build record. This crate answers them from a file. The same engine
that a platform hosts over HTTP therefore runs from a laptop, a container, or a test with the
same semantics: checkpoints, exactly-once output, stateful windows and joins.
use std::sync::atomic::AtomicBool;
use std::sync::Arc;
use fv_streams::Pipeline;
let stop = Arc::new(AtomicBool::new(false)); // set it to stop cleanly (Ctrl-C does)
Pipeline::load("pipeline.toml")?.run(stop).await;A pipeline file names its datasets (Kafka topics), its transforms (the engine’s own step
vocabulary: inline steps, wasm/container compute, windowedAggregate, sessionAggregate,
streamJoin, topN, lastN) and optional engine settings. The pipeline’s name is its stable
identity: the consumer group fv-stream-<name>, the state topic fv-state-<name> and the
exactly-once transactional namespace all derive from it, so a restarted run restores its
checkpoints and fences its own zombies.
Modules§
- flight_
sql - Flight-SQL statement submission (5d-ii): a Flight control endpoint where a client submits a 5c SQL statement and gets back an Arrow result stream.
- serve
- THE OBSERVABILITY SURFACE for an orchestrator (
fv-streams run --serve <addr>orSTREAM_SERVE=<addr>): one small HTTP listener beside the run.
Structs§
- File
Control Plane - The file-defined host: the engine’s
ControlPlaneanswered from aPipelinedefinition. Heartbeat = run until the stop flag; the build record prints as a status line (there is no server to send it to). - Pipeline
- A pipeline definition loaded from a TOML file.
Constants§
- SERVE
STREAM_SERVE: the runner’s observability listener (--serve).
Functions§
- settings
- Every setting this runner reads, by area — the engine’s, the connectors’, its own — what a
pipeline’s
[settings]and the environment are checked against, and whatdocs/settings.mdlists.