fv-streams-types
The contract of FusionVault Streams. It defines what a host, a connector and an operator each implement, and nothing else.
Its only dependencies are arrow, bytes, serde_json and async-trait.
A connector author depends on this crate alone, never on the engine.
What it holds
- The control-plane trait that a host implements.
- The source, sink and operator traits that tasks run.
- The decode policy for incoming messages, and the meta columns rows carry.
- The message formats: how a connector's messages carry rows, in JSON, Arrow IPC, raw values or CSV records.
- The settings reader.
- The vocabulary that steps, connectors and settings use to describe their options (
options,settings).
A source
A source fills an Out when it is polled, and records its position at a barrier:
use Arc;
use ;
use ;
use ;
let mut out = default;
let mut source = Counter ;
while let Again = source.poll
assert_eq!;
A message format
A connector reads the dataset's format once.
A sink turns each batch into payloads, and a source turns messages back into a batch.
use Arc;
use ;
use ;
use json;
let format = of.unwrap;
// a sink: a payload per row, read from the column where it lies
let batch = try_from_iter.unwrap;
let Rows = format.encode.unwrap else ;
assert_eq!;
// a source: the messages back into a batch
let mut decoder = format.decoder.unwrap;
let messages = rows.iter.enumerate.map.collect;
let decoded = decoder.decode_partition.unwrap;
assert_eq!;
A new format is one variant of MessageFormat, with a decoder and an encoder.
Every connector that carries messages then takes it.
Who uses it
- The engine,
fv-streams-engine, runs sources, operators and sinks as tasks. It sends the barriers and commits the epochs. - Every connector in
fv-streams-connectorsimplements these traits. fv-streams-sqlcompiles SQL to theTopologydefined here.
Licence
Apache 2.0.