fv-streams-types 0.9.0

The contract of FusionVault Streams: the control-plane seam, the task-facing source/sink/operator contract, the decode policy and meta columns, the settings reader, the option vocabulary steps and connectors describe themselves with — what a connector, an operator or a host depends on, and nothing more.
Documentation

fv-streams-types

The contract of FusionVault Streams. It defines what a host, a connector and an operator each implement, and nothing else.

Its only dependencies are arrow, bytes, serde_json and async-trait. A connector author depends on this crate alone, never on the engine.

What it holds

  • The control-plane trait that a host implements.
  • The source, sink and operator traits that tasks run.
  • The decode policy for incoming messages, and the meta columns rows carry.
  • The message formats: how a connector's messages carry rows, in JSON, Arrow IPC, raw values or CSV records.
  • The settings reader.
  • The vocabulary that steps, connectors and settings use to describe their options (options, settings).

A source

A source fills an Out when it is polled, and records its position at a barrier:

use std::sync::Arc;
use arrow::array::{Int64Array, RecordBatch};
use arrow::datatypes::{DataType, Field, Schema};
use fv_streams_types::{position_record, Out, Poll, Source};

struct Counter { next: i64, until: i64 }

impl Source for Counter {
    fn poll(&mut self, out: &mut Out) -> Poll {
        if self.next >= self.until {
            return Poll::Done;
        }
        let schema = Arc::new(Schema::new(vec![Field::new("n", DataType::Int64, false)]));
        out.push(RecordBatch::try_new(schema, vec![Arc::new(Int64Array::from(vec![self.next]))]).unwrap());
        self.next += 1;
        Poll::Again
    }
    fn on_barrier(&mut self, _epoch: u64) -> Vec<u8> {
        position_record("counter", 0, self.next)
    }
}

let mut out = Out::default();
let mut source = Counter { next: 0, until: 3 };
while let Poll::Again = source.poll(&mut out) {}
assert_eq!(out.batches.len(), 3);

A message format

A connector reads the dataset's format once. A sink turns each batch into payloads, and a source turns messages back into a batch.

use std::sync::Arc;
use arrow::array::{AsArray, RecordBatch, StringArray};
use fv_streams_types::format::{MessageFormat, Payloads};
use serde_json::json;

let format = MessageFormat::of("lines", &json!({ "format": "raw" })).unwrap();

// a sink: a payload per row, read from the column where it lies
let batch = RecordBatch::try_from_iter(vec![("line", Arc::new(StringArray::from(vec!["a", "b"])) as _)]).unwrap();
let Payloads::Rows(rows) = format.encode(&batch, usize::MAX).unwrap() else { unreachable!() };
assert_eq!(rows.get(0), b"a");

// a source: the messages back into a batch
let mut decoder = format.decoder(Default::default(), Default::default()).unwrap();
let messages = rows.iter().enumerate().map(|(i, p)| (None, i as i64, p.to_vec())).collect();
let decoded = decoder.decode_partition(messages).unwrap();
assert_eq!(decoded.batch.column(0).as_string::<i32>().value(1), "b");

A new format is one variant of MessageFormat, with a decoder and an encoder. Every connector that carries messages then takes it.

Who uses it

  • The engine, fv-streams-engine, runs sources, operators and sinks as tasks. It sends the barriers and commits the epochs.
  • Every connector in fv-streams-connectors implements these traits.
  • fv-streams-sql compiles SQL to the Topology defined here.

Licence

Apache 2.0.