fv-streams 0.8.0

The FusionVault Streams standalone runner: run a stream pipeline defined in one TOML file (steps or SQL) with no control plane — every connector built in, the Nexmark event generator, checkpoints to a local directory or an object store, scale-out across workers.
Documentation
//! Run a stream pipeline from one TOML file, with no control plane anywhere.
//!
//! The engine (`fv-streams-engine`) asks its host four questions through the
//! [`ControlPlane`](fv_streams_engine::ControlPlane) trait: what is the pipeline's topology, which topic backs a dataset, should I
//! keep running, and here is my build record. This crate answers them from a file. The same engine
//! that a platform hosts over HTTP therefore runs from a laptop, a container, or a test with the
//! same semantics: checkpoints, exactly-once output, stateful windows and joins.
//!
//! ```no_run
//! use std::sync::atomic::AtomicBool;
//! use std::sync::Arc;
//! use fv_streams::Pipeline;
//!
//! # tokio::runtime::Runtime::new().unwrap().block_on(async {
//! let stop = Arc::new(AtomicBool::new(false));      // set it to stop cleanly (Ctrl-C does)
//! Pipeline::load("pipeline.toml")?.run(stop).await;
//! # Ok::<(), String>(())
//! # });
//! ```
//!
//! A pipeline file names its datasets (Kafka topics), its transforms (the engine's own step
//! vocabulary: inline steps, `wasm`/`container` compute, `windowedAggregate`, `sessionAggregate`,
//! `streamJoin`, `topN`, `lastN`) and optional engine settings. The pipeline's `name` is its stable
//! identity: the consumer group `fv-stream-<name>`, the state topic `fv-state-<name>` and the
//! exactly-once transactional namespace all derive from it, so a restarted run restores its
//! checkpoints and fences its own zombies.

/// Every Rust code block in the README is compiled and run as a doctest.
#[cfg(doctest)]
#[doc = include_str!("../README.md")]
pub struct ReadmeDoctests;

use fv_streams_types::settings::{Section, Setting};

pub mod flight_sql;
pub mod serve;

mod control_plane;
mod pipeline;

pub use control_plane::FileControlPlane;
pub use pipeline::Pipeline;

/// `STREAM_SERVE`: the runner's observability listener (`--serve`).
pub const SERVE: Setting = Setting {
    key: "STREAM_SERVE",
    kind: fv_streams_types::options::Kind::Text,
    default: "",
    doc: "Serve `/health`, `/ready`, `/status` and `/metrics` on this address, `host:port`. Set by `--serve`. Empty: not served.",
};

static RUNNER: Section = Section {
    title: "The runner",
    intro: "The `fv-streams` command's own settings.",
    settings: &[SERVE],
};

/// Every setting this runner reads, by area — the engine's, the connectors', its own — what a
/// pipeline's `[settings]` and the environment are checked against, and what `docs/settings.md`
/// lists.
pub fn settings() -> Vec<&'static Section> {
    fv_streams_engine::settings::SECTIONS
        .iter()
        .chain(fv_streams_connectors::settings())
        .chain([&RUNNER])
        .collect()
}

#[cfg(test)]
mod tests;