fv-streams 0.9.0

The FusionVault Streams standalone runner: run a stream pipeline defined in one TOML file (steps or SQL) with no control plane — every connector built in, the Nexmark event generator, checkpoints to a local directory or an object store, scale-out across workers.
Documentation
# fv-streams

The runner for [FusionVault Streams](https://github.com/FusionVault/streams).
A pipeline is one TOML file, written as steps or as SQL, and the runner runs it with one command.
Every connector is built in, so nothing else needs to be running.

[Documentation](https://docs.fusionvault.net/streams/docs) ·
[Reference](#reference) ·
[Discord](https://discord.gg/fhG6Qcjqh)

## Install

```bash
cargo install fv-streams
```

The Streams repository builds with one codegen unit and thin LTO, which makes every query cheaper.
To install the same build:

```bash
CARGO_PROFILE_RELEASE_CODEGEN_UNITS=1 CARGO_PROFILE_RELEASE_LTO=thin cargo install fv-streams
```

## A first pipeline

This pipeline counts the bids on each auction, every 10 seconds.
The built-in Nexmark generator makes the events, so it needs nothing else to run.

```toml
name = "hot-auctions"

[datasets.events]
connector = "nexmark"
inline = true
events = 100000

[datasets.out]
sink = "stdout"

[[transforms]]
inputs = ["events"]
output = "out"

  [[transforms.steps]]
  op = "filter"
  expression = 'type == "bid"'

  [[transforms.steps]]
  op = "windowedAggregate"
  timeColumn = "dateTime"
  windowMs = 10000
  groupBy = ["auction"]
  aggs = [{ op = "count", alias = "bids" }]
```

Save it as `hot-auctions.toml`, and run it:

```bash
fv-streams run hot-auctions.toml
```

Each result prints as one line of JSON.

- A keyed step such as `windowedAggregate` first sends every row with the same key to the same
  task, so each auction gets one count.
- `name` is the pipeline's identity. Its checkpoints, its Kafka consumer group and its exactly-once
  records all use it, so a restart picks up where the last run stopped.

## Commands

| Command | What it does |
|---|---|
| `fv-streams run <pipeline.toml>` | Runs a pipeline until it ends or you stop it. |
| `fv-streams run --workers <n> <pipeline.toml>` | Leads a cluster of `n` workers, this one included. |
| `fv-streams run --join <leader> <pipeline.toml>` | Joins a cluster as a worker. |
| `fv-streams run --serve <addr> <pipeline.toml>` | Also serves `/health`, `/ready`, `/status` and `/metrics`. |
| `fv-streams flight-sql [--host <h>] [--port <n>] <catalog.toml>` | Answers `SELECT` queries over Arrow Flight, on the datasets the catalog declares. |
| `fv-streams probe <url>` | Exits 0 when `<url>` answers with a 2xx status. Use it as a container's health check. |

## Reference

The reference is generated from the code, so it says exactly what the runner does.

- [Steps]https://github.com/FusionVault/streams/blob/main/docs/reference/steps.md: every step and
  aggregation, every option with its default, and every output's columns.
- [SQL]https://github.com/FusionVault/streams/blob/main/docs/reference/sql.md: the same pipelines,
  written as `INSERT INTO … SELECT …`.
- [Functions]https://github.com/FusionVault/streams/blob/main/docs/reference/functions.md: every
  function an expression can call, with examples a test evaluates.
- [Connectors]https://github.com/FusionVault/streams/blob/main/docs/reference/connectors.md: every
  source and sink, its options, and what it guarantees.
- [Settings]https://github.com/FusionVault/streams/blob/main/docs/settings.md: every `STREAM_*`
  setting.

A key that a pipeline file does not take is refused when the pipeline starts.
That covers step options, dataset keys and settings, and the error names the closest key you might
have meant.

## As a library

```rust,no_run
use std::sync::{atomic::AtomicBool, Arc};
use fv_streams::Pipeline;

# tokio::runtime::Runtime::new().unwrap().block_on(async {
let stop = Arc::new(AtomicBool::new(false));
Pipeline::load("pipeline.toml")?.run(stop).await?;
# Ok::<(), String>(())
# });
```

## Licence

Apache 2.0.