fv-streams
The runner for FusionVault Streams. A pipeline is one TOML file, written as steps or as SQL, and the runner runs it with one command. Every connector is built in, so nothing else needs to be running.
Documentation · Reference · Discord
Install
The Streams repository builds with one codegen unit and thin LTO, which makes every query cheaper. To install the same build:
CARGO_PROFILE_RELEASE_CODEGEN_UNITS=1 CARGO_PROFILE_RELEASE_LTO=thin
A first pipeline
This pipeline counts the bids on each auction, every 10 seconds. The built-in Nexmark generator makes the events, so it needs nothing else to run.
= "hot-auctions"
[]
= "nexmark"
= true
= 100000
[]
= "stdout"
[[]]
= ["events"]
= "out"
[[]]
= "filter"
= 'type == "bid"'
[[]]
= "windowedAggregate"
= "dateTime"
= 10000
= ["auction"]
= [{ = "count", = "bids" }]
Save it as hot-auctions.toml, and run it:
Each result prints as one line of JSON.
- A keyed step such as
windowedAggregatefirst sends every row with the same key to the same task, so each auction gets one count. nameis the pipeline's identity. Its checkpoints, its Kafka consumer group and its exactly-once records all use it, so a restart picks up where the last run stopped.
Commands
| Command | What it does |
|---|---|
fv-streams run <pipeline.toml> |
Runs a pipeline until it ends or you stop it. |
fv-streams run --workers <n> <pipeline.toml> |
Leads a cluster of n workers, this one included. |
fv-streams run --join <leader> <pipeline.toml> |
Joins a cluster as a worker. |
fv-streams run --serve <addr> <pipeline.toml> |
Also serves /health, /ready, /status and /metrics. |
fv-streams flight-sql [--host <h>] [--port <n>] <catalog.toml> |
Answers SELECT queries over Arrow Flight, on the datasets the catalog declares. |
fv-streams probe <url> |
Exits 0 when <url> answers with a 2xx status. Use it as a container's health check. |
Reference
The reference is generated from the code, so it says exactly what the runner does.
- Steps: every step and aggregation, every option with its default, and every output's columns.
- SQL: the same pipelines,
written as
INSERT INTO … SELECT …. - Functions: every function an expression can call, with examples a test evaluates.
- Connectors: every source and sink, its options, and what it guarantees.
- Settings: every
STREAM_*setting.
A key that a pipeline file does not take is refused when the pipeline starts. That covers step options, dataset keys and settings, and the error names the closest key you might have meant.
As a library
use ;
use Pipeline;
# new.unwrap.block_on;
Licence
Apache 2.0.