fv-streams 0.2.0

The FusionVault Streams standalone runner: run a stream pipeline defined in one TOML file against Kafka or Redpanda with no control plane, including the built-in Nexmark event generator.
Documentation

fv-streams

The FusionVault Streams standalone runner: a stream pipeline is one TOML file, run against Kafka or Redpanda with no control plane anywhere.

cargo install fv-streams
fv-streams run pipeline.toml        # Ctrl-C stops it cleanly (final checkpoint, flush, commit)
name = "hot-bids"

[brokers]
default = "localhost:9092"

[datasets.bids]
topic = "bids"

[datasets.hot]
topic = "hot-bids"

[[transforms]]
inputs = ["bids"]
output = "hot"

  [[transforms.steps]]
  op = "filter"
  expression = "price > 100"

  [[transforms.steps]]
  op = "windowedAggregate"
  windowMs = 10000
  allowedLatenessMs = 2000
  groupBy = ["auction"]
  aggs = [{ op = "count", column = "", alias = "bids" }]

The pipeline's name is its identity: consumer group, state topic and exactly-once namespace all derive from it, so a restarted run restores its checkpoints.

Arrow & Flight edges

For an Arrow-speaking peer, an output dataset can skip the JSON encode entirely:

[datasets.out]
sink = "arrow"                 # write Arrow IPC files, one per sink task ({path}-part{t}.arrow)
path = "out"
# — or —
sink = "flight"                # serve the output as a live Arrow Flight do_get stream
port = 8815                     # one server per sink task at port + t (host defaults to 127.0.0.1)

A connector = "arrow" source reads Arrow IPC back in the same way. Both bypass the JSON decode/encode the Kafka edges carry; the measured tax is in bench/results/arrow-edge-tax-*. With STREAM_EXACTLY_ONCE=1 these sinks are commit-gated (output visible only when its epoch commits), matching the Kafka transactional sink; the default is at-least-once.

Flight-SQL — submit a SQL statement and read its Arrow result over Flight:

fv-streams flight-sql [--host <h>] [--port <n>] <catalog.toml>

<catalog.toml> declares only the datasets (and [settings]) statements read. A client sends a query Flight action carrying a SELECT …; the server compiles it, launches the pipeline with a Flight output, and returns that endpoint's grpc://… URL to read with an ordinary Flight do_get.

As a library

use std::sync::{atomic::AtomicBool, Arc};
use fv_streams::Pipeline;

# tokio::runtime::Runtime::new().unwrap().block_on(async {
let stop = Arc::new(AtomicBool::new(false));
Pipeline::load("pipeline.toml")?.run(stop).await?;
# Ok::<(), String>(())
# });

Nexmark built in

A dataset with connector = "nexmark" gets a deterministic generator of the standard Nexmark event mix (persons, auctions, bids) feeding its topic, so the whole benchmark suite runs from files: see the repository's bench/.

Apache-2.0.