fv-streams-sql 0.9.0

The SQL front door for FusionVault Streams: a streaming SQL statement parses to a small typed logical plan and lowers to the same topology the TOML frontend produces — one runtime, two front doors. No DataFusion planner, no fork; expressions compile through the value dialect as the steps already do.
Documentation

fv-streams-sql

The SQL front door of FusionVault Streams.

It compiles a streaming SQL statement to the same stages and steps that a TOML pipeline describes. The engine runs them the same way, so SQL gets checkpoints, exactly-once output and scale-out with no extra cost.

let topology = fv_streams_sql::compile(
    "INSERT INTO out SELECT auction, price * 0.908 AS eur FROM bids WHERE type = 'bid'",
)?;

In a pipeline

A pipeline file declares its datasets as usual, and puts the statement in sql instead of [[transforms]]:

name = "enrich"

sql = "INSERT INTO out SELECT auction, price * 0.908 AS eur FROM bids WHERE type = 'bid'"

[brokers]
default = "localhost:9092"

[datasets.bids]
topic = "bids"

[datasets.out]
topic = "out"

What SQL it supports

The SQL reference lists every statement form, what each one becomes, how expressions translate, and what is refused, with the message. It is generated from fv_streams_sql::reference. Each example on it is compiled to write the page, and a test fails if one stops compiling to the steps its form promises.

Some multi-stage statements, each checked against the same query written in TOML:

  • q13 computes a key in a WITH clause, then joins it to a lookup table.
  • q5 counts in sliding windows in a WITH clause, then keeps the top auction per window with ROW_NUMBER() <= 1.
  • q4 uses four WITH clauses: two filtered streams, their interval join, and a max per auction. It then takes an avg per category. That is five stages, with two windowed aggregates one after the other.

How it works

  • Its own plan, not a database's. The statement is parsed with sqlparser, then turned into a small typed plan that models streaming. There is no DataFusion logical plan and no fork. The errors speak about streams, and the plan is where an optimiser can plug in later.
  • One expression language. SQL expressions become the same expression strings the TOML steps use, compiled through fv-value-datafusion in the same way. So the two front doors cannot drift apart.

Licence

Apache 2.0.