fv-streams
The FusionVault Streams runner: a stream pipeline is one
TOML file — steps or SQL — run with no control plane. Every connector is built in (Kafka, NATS,
files and object stores, Iceberg, database tables, Arrow Flight, HTTP, WebSocket, the Nexmark
generator); checkpoints go to a local directory or an object store; --workers / --join spread a
pipeline across machines.
Built from crates.io, the runner gets Cargo's default release profile. The one this repository builds
with (one codegen unit, thin LTO) is cheaper on every query:
CARGO_PROFILE_RELEASE_CODEGEN_UNITS=1 CARGO_PROFILE_RELEASE_LTO=thin cargo install fv-streams.
A first pipeline
Nothing to install beyond the runner: the built-in Nexmark generator makes the events, and the results print as JSON lines.
= "hot-auctions"
[]
= "nexmark"
= true
= 100000
[]
= "stdout"
[[]]
= ["events"]
= "out"
[[]]
= "filter"
= 'type == "bid"'
[[]]
= "windowedAggregate"
= "dateTime"
= 10000
= ["auction"]
= [{ = "count", = "bids" }]
The pipeline's name is its identity: the consumer group, the checkpoint location and the
exactly-once namespace derive from it, so a restarted run restores its checkpoints.
Reference
Generated from the code, so they say what the runner does:
- Steps — every step and aggregation, every option with its default, every output's columns.
- SQL — the same pipelines
as
sql = "INSERT INTO … SELECT …". - Functions — every function an expression can call, with examples a test evaluates.
- Connectors — every source and sink, its options and what it guarantees.
- Settings — every
STREAM_*setting, by area.
A key a pipeline file does not take — a step's, a dataset's, a setting — is refused when it starts, with the closest one it might mean.
Commands
fv-streams run [--workers <n>] [--join <leader>] [--serve <addr>] <pipeline.toml>
fv-streams flight-sql [--host <h>] [--port <n>] <catalog.toml>
fv-streams probe <url>
runruns one pipeline until it ends or is stopped.--workers <n>makes this process the leader of a cluster ofnworkers, itself included;--join <leader>joins one;--serve <addr>serves/health,/ready,/statusand/metrics.flight-sqlserves a Flight endpoint: a client sends aSELECT …over the datasets<catalog.toml>declares and reads the Arrow result with an ordinary Flightdo_get.probeexits 0 when<url>answers 2xx: a container's health check.
As a library
use ;
use Pipeline;
# new.unwrap.block_on;
Apache-2.0.