# fv-streams
The runner for [FusionVault Streams](https://github.com/FusionVault/streams).
A pipeline is one TOML file, written as steps or as SQL, and the runner runs it with one command.
Every connector is built in, so nothing else needs to be running.
[Documentation](https://docs.fusionvault.net/streams/docs) ·
[Reference](#reference) ·
[Discord](https://discord.gg/fhG6Qcjqh)
## Install
```bash
cargo install fv-streams
```
The Streams repository builds with one codegen unit and thin LTO, which makes every query cheaper.
To install the same build:
```bash
CARGO_PROFILE_RELEASE_CODEGEN_UNITS=1 CARGO_PROFILE_RELEASE_LTO=thin cargo install fv-streams
```
## A first pipeline
This pipeline counts the bids on each auction, every 10 seconds.
The built-in Nexmark generator makes the events, so it needs nothing else to run.
```toml
name = "hot-auctions"
[datasets.events]
connector = "nexmark"
inline = true
events = 100000
[datasets.out]
sink = "stdout"
[[transforms]]
inputs = ["events"]
output = "out"
[[transforms.steps]]
op = "filter"
expression = 'type == "bid"'
[[transforms.steps]]
op = "windowedAggregate"
timeColumn = "dateTime"
windowMs = 10000
groupBy = ["auction"]
aggs = [{ op = "count", alias = "bids" }]
```
Save it as `hot-auctions.toml`, and run it:
```bash
fv-streams run hot-auctions.toml
```
Each result prints as one line of JSON.
- A keyed step such as `windowedAggregate` first sends every row with the same key to the same
task, so each auction gets one count.
- `name` is the pipeline's identity. Its checkpoints, its Kafka consumer group and its exactly-once
records all use it, so a restart picks up where the last run stopped.
## Commands
| `fv-streams run <pipeline.toml>` | Runs a pipeline until it ends or you stop it. |
| `fv-streams run --workers <n> <pipeline.toml>` | Leads a cluster of `n` workers, this one included. |
| `fv-streams run --join <leader> <pipeline.toml>` | Joins a cluster as a worker. |
| `fv-streams run --serve <addr> <pipeline.toml>` | Also serves `/health`, `/ready`, `/status` and `/metrics`. |
| `fv-streams flight-sql [--host <h>] [--port <n>] <catalog.toml>` | Answers `SELECT` queries over Arrow Flight, on the datasets the catalog declares. |
| `fv-streams probe <url>` | Exits 0 when `<url>` answers with a 2xx status. Use it as a container's health check. |
## Reference
The reference is generated from the code, so it says exactly what the runner does.
- [Steps](https://github.com/FusionVault/streams/blob/main/docs/reference/steps.md): every step and
aggregation, every option with its default, and every output's columns.
- [SQL](https://github.com/FusionVault/streams/blob/main/docs/reference/sql.md): the same pipelines,
written as `INSERT INTO … SELECT …`.
- [Functions](https://github.com/FusionVault/streams/blob/main/docs/reference/functions.md): every
function an expression can call, with examples a test evaluates.
- [Connectors](https://github.com/FusionVault/streams/blob/main/docs/reference/connectors.md): every
source and sink, its options, and what it guarantees.
- [Settings](https://github.com/FusionVault/streams/blob/main/docs/settings.md): every `STREAM_*`
setting.
A key that a pipeline file does not take is refused when the pipeline starts.
That covers step options, dataset keys and settings, and the error names the closest key you might
have meant.
## As a library
```rust,no_run
use std::sync::{atomic::AtomicBool, Arc};
use fv_streams::Pipeline;
# tokio::runtime::Runtime::new().unwrap().block_on(async {
let stop = Arc::new(AtomicBool::new(false));
Pipeline::load("pipeline.toml")?.run(stop).await?;
# Ok::<(), String>(())
# });
```
## Licence
Apache 2.0.