fv-streams
The FusionVault Streams standalone runner: a stream pipeline is one TOML file, run against Kafka or Redpanda with no control plane anywhere.
= "hot-bids"
[]
= "localhost:9092"
[]
= "bids"
[]
= "hot-bids"
[[]]
= ["bids"]
= "hot"
[[]]
= "filter"
= "price > 100"
[[]]
= "windowedAggregate"
= 10000
= 2000
= ["auction"]
= [{ = "count", = "", = "bids" }]
The pipeline's name is its identity: consumer group, state topic and exactly-once namespace all
derive from it, so a restarted run restores its checkpoints.
Arrow & Flight edges
For an Arrow-speaking peer, an output dataset can skip the JSON encode entirely:
[]
= "arrow" # write Arrow IPC files, one per sink task ({path}-part{t}.arrow)
= "out"
# — or —
= "flight" # serve the output as a live Arrow Flight do_get stream
= 8815 # one server per sink task at port + t (host defaults to 127.0.0.1)
A connector = "arrow" source reads Arrow IPC back in the same way. Both bypass the JSON
decode/encode the Kafka edges carry; the measured tax is in bench/results/arrow-edge-tax-*. With
STREAM_EXACTLY_ONCE=1 these sinks are commit-gated (output visible only when its epoch commits),
matching the Kafka transactional sink; the default is at-least-once.
Flight-SQL — submit a SQL statement and read its Arrow result over Flight:
fv-streams flight-sql [--host <h>] [--port <n>] <catalog.toml>
<catalog.toml> declares only the datasets (and [settings]) statements read. A client sends a
query Flight action carrying a SELECT …; the server compiles it, launches the pipeline with a
Flight output, and returns that endpoint's grpc://… URL to read with an ordinary Flight do_get.
As a library
use ;
use Pipeline;
# new.unwrap.block_on;
Nexmark built in
A dataset with connector = "nexmark" gets a deterministic generator of the standard Nexmark
event mix (persons, auctions, bids) feeding its topic, so the whole benchmark suite runs from
files: see the repository's bench/.
Apache-2.0.