fv-streams 0.7.0

The FusionVault Streams standalone runner: run a stream pipeline defined in one TOML file (steps or SQL) with no control plane — every connector built in, the Nexmark event generator, checkpoints to a local directory or an object store, scale-out across workers.
Documentation

fv-streams

The FusionVault Streams runner: a stream pipeline is one TOML file — steps or SQL — run with no control plane. Every connector is built in (Kafka, NATS, files and object stores, Iceberg, database tables, Arrow Flight, HTTP, WebSocket, the Nexmark generator); checkpoints go to a local directory or an object store; --workers / --join spread a pipeline across machines.

cargo install fv-streams
fv-streams run pipeline.toml        # Ctrl-C stops it in order: a final checkpoint, flush, commit

Built from crates.io, the runner gets Cargo's default release profile. The one this repository builds with (one codegen unit, thin LTO) is cheaper on every query: CARGO_PROFILE_RELEASE_CODEGEN_UNITS=1 CARGO_PROFILE_RELEASE_LTO=thin cargo install fv-streams.

A first pipeline

Nothing to install beyond the runner: the built-in Nexmark generator makes the events, and the results print as JSON lines.

name = "hot-auctions"

[datasets.events]
connector = "nexmark"
inline = true
events = 100000

[datasets.out]
sink = "stdout"

[[transforms]]
inputs = ["events"]
output = "out"

  [[transforms.steps]]
  op = "filter"
  expression = 'type == "bid"'

  [[transforms.steps]]
  op = "windowedAggregate"
  timeColumn = "dateTime"
  windowMs = 10000
  groupBy = ["auction"]
  aggs = [{ op = "count", alias = "bids" }]

The pipeline's name is its identity: the consumer group, the checkpoint location and the exactly-once namespace derive from it, so a restarted run restores its checkpoints.

Reference

Generated from the code, so they say what the runner does:

  • Steps — every step and aggregation, every option with its default, every output's columns.
  • SQL — the same pipelines as sql = "INSERT INTO … SELECT …".
  • Functions — every function an expression can call, with examples a test evaluates.
  • Connectors — every source and sink, its options and what it guarantees.
  • Settings — every STREAM_* setting, by area.

A key a pipeline file does not take — a step's, a dataset's, a setting — is refused when it starts, with the closest one it might mean.

Commands

fv-streams run [--workers <n>] [--join <leader>] [--serve <addr>] <pipeline.toml>
fv-streams flight-sql [--host <h>] [--port <n>] <catalog.toml>
fv-streams probe <url>
  • run runs one pipeline until it ends or is stopped. --workers <n> makes this process the leader of a cluster of n workers, itself included; --join <leader> joins one; --serve <addr> serves /health, /ready, /status and /metrics.
  • flight-sql serves a Flight endpoint: a client sends a SELECT … over the datasets <catalog.toml> declares and reads the Arrow result with an ordinary Flight do_get.
  • probe exits 0 when <url> answers 2xx: a container's health check.

As a library

use std::sync::{atomic::AtomicBool, Arc};
use fv_streams::Pipeline;

# tokio::runtime::Runtime::new().unwrap().block_on(async {
let stop = Arc::new(AtomicBool::new(false));
Pipeline::load("pipeline.toml")?.run(stop).await?;
# Ok::<(), String>(())
# });

Apache-2.0.