fv-streams-sql
The SQL front door of FusionVault Streams.
It compiles a streaming SQL statement to the same stages and steps that a TOML pipeline describes. The engine runs them the same way, so SQL gets checkpoints, exactly-once output and scale-out with no extra cost.
let topology = compile?;
In a pipeline
A pipeline file declares its datasets as usual, and puts the statement in sql instead of
[[transforms]]:
= "enrich"
= "INSERT INTO out SELECT auction, price * 0.908 AS eur FROM bids WHERE type = 'bid'"
[]
= "localhost:9092"
[]
= "bids"
[]
= "out"
What SQL it supports
The SQL reference lists
every statement form, what each one becomes, how expressions translate, and what is refused, with
the message.
It is generated from fv_streams_sql::reference.
Each example on it is compiled to write the page, and a test fails if one stops compiling to the
steps its form promises.
Some multi-stage statements, each checked against the same query written in TOML:
- q13 computes a key in a
WITHclause, then joins it to a lookup table. - q5 counts in sliding windows in a
WITHclause, then keeps the top auction per window withROW_NUMBER() <= 1. - q4 uses four
WITHclauses: two filtered streams, their interval join, and amaxper auction. It then takes anavgper category. That is five stages, with two windowed aggregates one after the other.
How it works
- Its own plan, not a database's. The statement is parsed with
sqlparser, then turned into a small typed plan that models streaming. There is no DataFusion logical plan and no fork. The errors speak about streams, and the plan is where an optimiser can plug in later. - One expression language. SQL expressions become the same expression strings the TOML steps
use, compiled through
fv-value-datafusionin the same way. So the two front doors cannot drift apart.
Licence
Apache 2.0.