Expand description
The CONTRACT of FusionVault Streams — what every crate agrees on, and nothing more:
seam: the control-plane seam a host implements (ControlPlane,Topology/StageDef,Binding) and the source/sink factories every connector and host implements (InlineSource,OutputSink, withSourceCtx/SinkCtx);contract: the task-facing contract a connector or operator implements (Source,Sink,Operator,Out,Poll, the snapshot types,EpochStagingfor exactly-once sinks);decode: the source decode policy (declared or inferred schema, bad-data handling) and the meta columns;format: the message formats (JSON, Arrow IPC, raw, CSV): how a connector’s messages carry rows, with a decoder and an encoder each;settings: theSTREAM_*settings reader;time: what a column’s values mean as event time (epoch ms), for every reader of one.
The engine (the runtime, the planner), the connectors and the SQL front door are built on top of this crate; a connector author, an embedder or a SQL tool depends on this crate alone.
Re-exports§
Modules§
- contract
- The task-facing contract: what a source, a sink or an operator implements, and what
the runtime hands it — an
Outto push batches into, a barrier to record a position at, a commit to act on. The runtime that schedules tasks over edges lives in the engine; a connector, an operator or an embedder depends on this crate alone. - decode
- COLUMNAR DECODE at the source edge: Kafka messages → one Arrow
RecordBatchper partition per poll, under the source’s schema and bad-data policy. - field
- A DECLARED FIELD, in one place: the type grammar a dataset’s
schemais written in, and its Arrow form. Every source reads its declared columns through here. - format
- MESSAGE FORMATS: how a connector’s messages carry rows (
format = …). One seam for every connector that reads or writes messages, kept in the contract so a connector outside this repo plugs in the same way. A source reads each message with the format’s decoder. A sink turns each batch into the format’s payloads. A transport that carries many messages in one body (an HTTP response, a webhook’s POST, the lines of stdout) frames them by the format’s rule. The broker or the transport never reads a message: to Kafka, NATS or HTTP it is bytes. - options
- Describing options: the vocabulary a step or a connector uses to say, once, what it accepts —
each option’s key, type, default, whether it is required and what it does. The same description
checks a pipeline file (
check_keys: an unknown key is refused with the closest right one) and renders the reference tables (Opt::row), so the two can never disagree. - seam
- The CONTROL-PLANE SEAM — everything the streaming engine needs from its host, and nothing more. The engine owns streaming semantics (specs, operators, watermarks, checkpoints, exactly-once); the host owns where pipelines come from and what datasets mean — which connector reads or writes each one. Two hosts exist by design: an application’s control plane (the contract every e2e mocks prove) and the standalone runner’s file-defined pipelines.
- settings
STREAM_*settings, each described once — its name, type, default and what it does — and read through that description, so the default lives in one place and the reference (docs/settings.md) is rendered from what the code reads. A crate describes the settings it reads asSections; a host gathers every crate’s and checks a pipeline’s[settings]table (check_table) and the environment (check_environment) against them before it starts.- time
- EVENT TIME: what a column’s values mean as a time, in one place. Every reader of an event-time column takes its times from here: a source stamping its watermark, a window, a session, a join.