fv-streams 0.9.0

The FusionVault Streams standalone runner: run a stream pipeline defined in one TOML file (steps or SQL) with no control plane — every connector built in, the Nexmark event generator, checkpoints to a local directory or an object store, scale-out across workers.
Documentation

fv-streams

The runner for FusionVault Streams. A pipeline is one TOML file, written as steps or as SQL, and the runner runs it with one command. Every connector is built in, so nothing else needs to be running.

Documentation · Reference · Discord

Install

cargo install fv-streams

The Streams repository builds with one codegen unit and thin LTO, which makes every query cheaper. To install the same build:

CARGO_PROFILE_RELEASE_CODEGEN_UNITS=1 CARGO_PROFILE_RELEASE_LTO=thin cargo install fv-streams

A first pipeline

This pipeline counts the bids on each auction, every 10 seconds. The built-in Nexmark generator makes the events, so it needs nothing else to run.

name = "hot-auctions"

[datasets.events]
connector = "nexmark"
inline = true
events = 100000

[datasets.out]
sink = "stdout"

[[transforms]]
inputs = ["events"]
output = "out"

  [[transforms.steps]]
  op = "filter"
  expression = 'type == "bid"'

  [[transforms.steps]]
  op = "windowedAggregate"
  timeColumn = "dateTime"
  windowMs = 10000
  groupBy = ["auction"]
  aggs = [{ op = "count", alias = "bids" }]

Save it as hot-auctions.toml, and run it:

fv-streams run hot-auctions.toml

Each result prints as one line of JSON.

  • A keyed step such as windowedAggregate first sends every row with the same key to the same task, so each auction gets one count.
  • name is the pipeline's identity. Its checkpoints, its Kafka consumer group and its exactly-once records all use it, so a restart picks up where the last run stopped.

Commands

Command What it does
fv-streams run <pipeline.toml> Runs a pipeline until it ends or you stop it.
fv-streams run --workers <n> <pipeline.toml> Leads a cluster of n workers, this one included.
fv-streams run --join <leader> <pipeline.toml> Joins a cluster as a worker.
fv-streams run --serve <addr> <pipeline.toml> Also serves /health, /ready, /status and /metrics.
fv-streams flight-sql [--host <h>] [--port <n>] <catalog.toml> Answers SELECT queries over Arrow Flight, on the datasets the catalog declares.
fv-streams probe <url> Exits 0 when <url> answers with a 2xx status. Use it as a container's health check.

Reference

The reference is generated from the code, so it says exactly what the runner does.

  • Steps: every step and aggregation, every option with its default, and every output's columns.
  • SQL: the same pipelines, written as INSERT INTO … SELECT ….
  • Functions: every function an expression can call, with examples a test evaluates.
  • Connectors: every source and sink, its options, and what it guarantees.
  • Settings: every STREAM_* setting.

A key that a pipeline file does not take is refused when the pipeline starts. That covers step options, dataset keys and settings, and the error names the closest key you might have meant.

As a library

use std::sync::{atomic::AtomicBool, Arc};
use fv_streams::Pipeline;

# tokio::runtime::Runtime::new().unwrap().block_on(async {
let stop = Arc::new(AtomicBool::new(false));
Pipeline::load("pipeline.toml")?.run(stop).await?;
# Ok::<(), String>(())
# });

Licence

Apache 2.0.