fv-streams-types 0.7.0

The contract of FusionVault Streams: the control-plane seam, the task-facing source/sink/operator contract, the decode policy and meta columns, the settings reader, the option vocabulary steps and connectors describe themselves with — what a connector, an operator or a host depends on, and nothing more.
Documentation

fv-streams-types

The contract of FusionVault Streams: the control-plane seam a host implements, the task-facing contract a connector or an operator implements, the source decode policy and the meta columns, and the vocabulary steps, connectors and settings describe their options with (options, settings). Nothing else — arrow, serde_json and async-trait are its only dependencies, so a connector author depends on this crate alone and never on the engine.

A source is a type that fills an Out when polled and records its position at a barrier:

use std::sync::Arc;
use arrow::array::{Int64Array, RecordBatch};
use arrow::datatypes::{DataType, Field, Schema};
use fv_streams_types::{position_record, Out, Poll, Source};

struct Counter { next: i64, until: i64 }

impl Source for Counter {
    fn poll(&mut self, out: &mut Out) -> Poll {
        if self.next >= self.until {
            return Poll::Done;
        }
        let schema = Arc::new(Schema::new(vec![Field::new("n", DataType::Int64, false)]));
        out.push(RecordBatch::try_new(schema, vec![Arc::new(Int64Array::from(vec![self.next]))]).unwrap());
        self.next += 1;
        Poll::Again
    }
    fn on_barrier(&mut self, _epoch: u64) -> Vec<u8> {
        position_record("counter", 0, self.next)
    }
}

let mut out = Out::default();
let mut source = Counter { next: 0, until: 3 };
while let Poll::Again = source.poll(&mut out) {}
assert_eq!(out.batches.len(), 3);

The engine (fv-streams-engine) runs sources, operators and sinks as tasks over edges, takes the barriers and commits the epochs; every connector (fv-streams-connectors) implements this contract; fv-streams-sql lowers SQL to the Topology defined here.

License

Apache-2.0.