Expand description
THE CONNECTORS of FusionVault Streams — one directory per connector, each behind its own
feature, all through the contract crate’s seams: a source is an
fv_streams_types::InlineSource, a sink an
fv_streams_types::OutputSink; the engine runs them as tasks and never knows
which is which. Adding a connector is a directory (mod.rs: the config parse, the Connector
registration and its description, Connector::about; source.rs / sink.rs: the
implementations), a feature line, and one entry in connectors. The description is what the
dataset check (check_dataset) accepts and what docs/reference/connectors.md shows.
connector = … / sink = … | directory | feature |
|---|---|---|
blackhole (sink) | blackhole | — |
file, arrow (source + sink) | file | — (s3 / gcs / azure for an object store) |
table (source) | table | sqlite / postgres / mysql |
nexmark (source) | nexmark | — |
iceberg (sink) | iceberg | iceberg |
flight (sink) | flight | flight |
kafka (source + sink) — the default when a dataset names neither | kafka | kafka |
nats (source + sink; JetStream) | nats | nats |
stdout (sink) — the minimum connector, the worked example | stdout | — |
http (polling source), webhook (sink) | http | http |
websocket (source: a live feed) | websocket | websocket |
Re-exports§
pub use fv_streams_formats as formats;
Modules§
- blackhole
sink = "blackhole": nowhere but the emitted counter — the sink of a measurement whose question is the engine, not the edge.- exactly_
once - EXACTLY-ONCE DELIVERY for any sink, in one place. A connector supplies an
EpochDestination— how one epoch’s rows reach its system, and (when that system can record it) which epoch reached it last — andsinkreturns theSinkthat runs the protocol around it: - fanout
- The reusable output fan-out — the core an Arrow-native output edge shares, independent of its
transport. A
BatchFanoutis aSinkthat strips the__fv_meta columns and publishes each data batch to abroadcastchannel; any number of readers subscribe and receive the sameOutputMsgstream. The Flight output sink wraps one behind a gRPC server; another output edge can wrap one behind its own wire, so “serve the output as a live Arrow stream” lives in one place. - file
connector = "file"/sink = "file"(and"arrow", the short form offormat = "arrow"): files and object stores at the edges — a source lists a directory or ans3://prefix and reads its files straight into batches (no JSON decode); a sink writes the output as immutable parts of onefv_streams_formatsformat under a committed manifest, in two phases aligned with the checkpoint, so a crash never loses a committed row and a restored run continues where it committed.partitionBywrites Hive-stylecol=value/directories;rollBytesrolls a part that grows past it;compressionapplies to Parquet.- messaging
- MESSAGE HELPERS shared by every connector that reads or writes messages (Kafka, NATS, HTTP,
WebSocket, stdout, and the next one): the message format, the drain that coalesces a poll into
one batch, the decode of a drained run into one Arrow batch per partition, the delivery rule, and
a row’s key on a broker. The message format itself (
format = …: its decoders, its encoders and how a body holds many messages) lives in the contract crate,fv_streams_types::format, and is re-exported here. Public: a connector written outside this crate gets all of it too. - nexmark
connector = "nexmark": the deterministic Nexmark generator — in-process (inline = true) as a source whose tasks generate straight into Arrow batches, or written to the dataset’s Kafka topic by [spawn] so the Kafka source reads it (the topic mode; featurekafka).- reference
- The connectors reference:
docs/reference/connectors.md, rendered from each connector’s own description (crate::Connector::about) — the same description the dataset check reads, so the page and the check can never disagree. The runner renders it with every connector built in, and a test there fails when the file is stale (make referencerewrites it). - stdout
sink = "stdout": the output on standard output, a row per line in the dataset’sformat(JSON lines by default, raw values or CSV records) — the smallest connector there is, and the worked example of what one needs: this file (the registration and the parse) and a sink.stderr = truewrites to standard error instead (a runner’s own lines go to stdout).- table
connector = "table": a bounded database table (SQLite / Postgres / MySQL through sqlx, each a feature) as the lookup join’s side input —tableorquery(the database’s own SQL: pushdown by construction),refreshMsre-reads on schedule. Its size estimate sizes the join’s distribution.
Structs§
- Connector
Doc - What a connector is and every option it takes, per role — the one description the dataset check
(
check_dataset) and the generated reference (docs/reference/connectors.md,reference) both read. - Ctx
- What a connector parses: the dataset’s name and its table from the pipeline definition, and the
pipeline’s
[brokers]table (Nullwhen it has none): the default brokers, and the connection options, for a connector that reaches Kafka. - RoleDoc
- One role of a connector (source or sink).
Constants§
- RENAMED
- Dataset options that were renamed,
(old, new): an old name is still accepted wherever its new name is, with a warning that names the new one; setting both is refused.
Traits§
- Connector
- One connector: the
connector = "…"spellings it reads as a source, thesink = "…"spellings it writes as a sink, and the parse of each from the dataset table into the seam’s kinds.
Functions§
- check_
dataset - Check a dataset’s keys against what its connectors take: every key must be
connector,sink, or an option of the source the dataset names (or the default’s) or of its sink — or the old name of one (RENAMED). An unknown key is refused with the closest right one; the error lists what the dataset takes, the roles it names first.Okholds a warning for each old name used. A dataset whose source or sink is not described is not checked. - connectors
- Every connector this build carries.
- resolve_
sink - The sink a dataset’s
sink = "…"names — or the default connector’s when it names none. - resolve_
source - The source a dataset’s
connector = "…"names — or the default connector’s when it names none (a build without the default names the gap).Errnames the dataset and what is wrong. - settings
- Every setting the connectors of this build read, by area: the shared source settings, then each connector’s, by title.