fv-streams 0.1.0

The FusionVault Streams standalone runner: run a stream pipeline defined in one TOML file against Kafka or Redpanda with no control plane, including the built-in Nexmark event generator.
Documentation

fv-streams

The FusionVault Streams standalone runner: a stream pipeline is one TOML file, run against Kafka or Redpanda with no control plane anywhere.

cargo install fv-streams
fv-streams run pipeline.toml        # Ctrl-C stops it cleanly (final checkpoint, flush, commit)
name = "hot-bids"

[brokers]
default = "localhost:9092"

[datasets.bids]
topic = "bids"

[datasets.hot]
topic = "hot-bids"

[[transforms]]
inputs = ["bids"]
output = "hot"

  [[transforms.steps]]
  op = "filter"
  expression = "price > 100"

  [[transforms.steps]]
  op = "windowedAggregate"
  windowMs = 10000
  allowedLatenessMs = 2000
  groupBy = ["auction"]
  aggs = [{ op = "count", column = "", alias = "bids" }]

The pipeline's name is its identity: consumer group, state topic and exactly-once namespace all derive from it, so a restarted run restores its checkpoints.

As a library

use std::sync::{atomic::AtomicBool, Arc};
use fv_streams::Pipeline;

# tokio::runtime::Runtime::new().unwrap().block_on(async {
let stop = Arc::new(AtomicBool::new(false));
Pipeline::load("pipeline.toml")?.run(stop).await?;
# Ok::<(), String>(())
# });

Nexmark built in

A dataset with connector = "nexmark" gets a deterministic generator of the standard Nexmark event mix (persons, auctions, bids) feeding its topic, so the whole benchmark suite runs from files: see the repository's bench/.

Apache-2.0.