Skip to main content

Crate fv_streams_types

Crate fv_streams_types 

Source
Expand description

The CONTRACT of FusionVault Streams — what every crate agrees on, and nothing more:

  • seam: the control-plane seam a host implements (ControlPlane, Topology/StageDef, Binding) and the source/sink factories every connector and host implements (InlineSource, OutputSink, with SourceCtx / SinkCtx);
  • contract: the task-facing contract a connector or operator implements (Source, Sink, Operator, Out, Poll, the snapshot types, EpochStaging for exactly-once sinks);
  • decode: the source decode policy (declared or inferred schema, bad-data handling) and the meta columns;
  • format: the message formats (JSON, Arrow IPC, raw, CSV): how a connector’s messages carry rows, with a decoder and an encoder each;
  • settings: the STREAM_* settings reader;
  • time: what a column’s values mean as event time (epoch ms), for every reader of one.

The engine (the runtime, the planner), the connectors and the SQL front door are built on top of this crate; a connector author, an embedder or a SQL tool depends on this crate alone.

Re-exports§

pub use contract::*;
pub use seam::*;

Modules§

contract
The task-facing contract: what a source, a sink or an operator implements, and what the runtime hands it — an Out to push batches into, a barrier to record a position at, a commit to act on. The runtime that schedules tasks over edges lives in the engine; a connector, an operator or an embedder depends on this crate alone.
decode
COLUMNAR DECODE at the source edge: Kafka messages → one Arrow RecordBatch per partition per poll, under the source’s schema and bad-data policy.
field
A DECLARED FIELD, in one place: the type grammar a dataset’s schema is written in, and its Arrow form. Every source reads its declared columns through here.
format
MESSAGE FORMATS: how a connector’s messages carry rows (format = …). One seam for every connector that reads or writes messages, kept in the contract so a connector outside this repo plugs in the same way. A source reads each message with the format’s decoder. A sink turns each batch into the format’s payloads. A transport that carries many messages in one body (an HTTP response, a webhook’s POST, the lines of stdout) frames them by the format’s rule. The broker or the transport never reads a message: to Kafka, NATS or HTTP it is bytes.
options
Describing options: the vocabulary a step or a connector uses to say, once, what it accepts — each option’s key, type, default, whether it is required and what it does. The same description checks a pipeline file (check_keys: an unknown key is refused with the closest right one) and renders the reference tables (Opt::row), so the two can never disagree.
seam
The CONTROL-PLANE SEAM — everything the streaming engine needs from its host, and nothing more. The engine owns streaming semantics (specs, operators, watermarks, checkpoints, exactly-once); the host owns where pipelines come from and what datasets mean — which connector reads or writes each one. Two hosts exist by design: an application’s control plane (the contract every e2e mocks prove) and the standalone runner’s file-defined pipelines.
settings
STREAM_* settings, each described once — its name, type, default and what it does — and read through that description, so the default lives in one place and the reference (docs/settings.md) is rendered from what the code reads. A crate describes the settings it reads as Sections; a host gathers every crate’s and checks a pipeline’s [settings] table (check_table) and the environment (check_environment) against them before it starts.
time
EVENT TIME: what a column’s values mean as a time, in one place. Every reader of an event-time column takes its times from here: a source stamping its watermark, a window, a session, a join.