fv-streams-sql 0.7.0

The SQL front door for FusionVault Streams: a streaming SQL statement parses to a small typed logical plan and lowers to the same topology the TOML frontend produces — one runtime, two front doors. No DataFusion planner, no fork; expressions compile through the value dialect as the steps already do.
Documentation

fv-streams-sql

The SQL front door for FusionVault Streams.

A streaming SQL statement parses to a small typed logical plan and lowers to the same topology the TOML frontend produces (its steps are the engine's own JSON vocabulary). One runtime, two front doors: SQL inherits checkpointing, the exchange, exactness and --join because it emits the identical steps a hand-written pipeline would.

let topo = fv_streams_sql::compile(
    "INSERT INTO out SELECT auction, price * 0.908 AS eur FROM bids WHERE type = 'bid'",
)?;

Design

  • No DataFusion LogicalPlan / physical layer, no fork. We lower the sqlparser AST to our own IR, which models streaming (DataFusion's logical plan does not); the errors are streaming-aware and ours, and the IR is the seam a future optimiser plugs into. (See design/…-phase5-… §10.)
  • Expressions reuse the value dialect. SQL scalar expressions unparse to the value-dialect strings the expression steps already carry, compiled through fv-value-datafusion at build time exactly as the TOML path does — one expression layer, so the two front doors cannot drift.

The supported SQL

Every statement form, what each one becomes, how expressions translate, and what is refused (with the message): docs/reference/sql.md, generated from fv_streams_sql::reference — each example on it is compiled to write it, and a test fails when an example stops compiling to the steps its form promises.

Multi-stage worked examples (each e2e-exact against its TOML twin's oracle): q13 — WITH bids AS (SELECT *, auction % N AS key … WHERE type='bid') SELECT * FROM bids JOIN side ON bids.key = side.key (a computed key, then the lookup join); q5 — a sliding windowed count CTE then a per-window ROW_NUMBER() <= 1; q4 — four CTEs (two filtered streams, their interval join, a per-auction max) then an avg per category, five stages with two nested windowed aggregates.

In a pipeline

A pipeline config declares its datasets as usual and puts the query in a sql field instead of [[transforms]]:

name = "enrich"
sql = "INSERT INTO out SELECT auction, price * 0.908 AS eur FROM bids WHERE type = 'bid'"
[brokers]
default = "localhost:9092"
[datasets.bids]
topic = "bids"
[datasets.out]
topic = "out"