Skip to main content

Crate fv_streams

Crate fv_streams 

Source
Expand description

Run a stream pipeline from one TOML file, with no control plane anywhere.

The engine (fv-streams-engine) asks its host four questions through the ControlPlane trait: what is the pipeline’s topology, which topic backs a dataset, should I keep running, and here is my build record. This crate answers them from a file. The same engine that a platform hosts over HTTP therefore runs from a laptop, a container, or a test with the same semantics: checkpoints, exactly-once output, stateful windows and joins.

use std::sync::atomic::AtomicBool;
use std::sync::Arc;
use fv_streams::Pipeline;

let stop = Arc::new(AtomicBool::new(false));      // set it to stop cleanly (Ctrl-C does)
Pipeline::load("pipeline.toml")?.run(stop).await;

A pipeline file names its datasets (Kafka topics), its transforms (the engine’s own step vocabulary: inline steps, wasm/container compute, windowedAggregate, sessionAggregate, streamJoin, topN, lastN) and optional engine settings. The pipeline’s name is its stable identity: the consumer group fv-stream-<name>, the state topic fv-state-<name> and the exactly-once transactional namespace all derive from it, so a restarted run restores its checkpoints and fences its own zombies.

Modules§

flight_sql
Flight-SQL statement submission (5d-ii): a Flight control endpoint where a client submits a 5c SQL statement and gets back an Arrow result stream.
serve
THE OBSERVABILITY SURFACE for an orchestrator (fv-streams run --serve <addr> or STREAM_SERVE=<addr>): one small HTTP listener beside the run.

Structs§

FileControlPlane
The file-defined host: the engine’s ControlPlane answered from a Pipeline definition. Heartbeat = run until the stop flag; the build record prints as a status line (there is no server to send it to).
Pipeline
A pipeline definition loaded from a TOML file.

Constants§

SERVE
STREAM_SERVE: the runner’s observability listener (--serve).

Functions§

settings
Every setting this runner reads, by area — the engine’s, the connectors’, its own — what a pipeline’s [settings] and the environment are checked against, and what docs/settings.md lists.