Skip to main content

Crate fv_streams_connectors

Crate fv_streams_connectors 

Source
Expand description

THE CONNECTORS of FusionVault Streams — one directory per connector, each behind its own feature, all through the contract crate’s seams: a source is an fv_streams_types::InlineSource, a sink an fv_streams_types::OutputSink; the engine runs them as tasks and never knows which is which. Adding a connector is a directory (mod.rs: the config parse, the Connector registration and its description, Connector::about; source.rs / sink.rs: the implementations), a feature line, and one entry in connectors. The description is what the dataset check (check_dataset) accepts and what docs/reference/connectors.md shows.

connector = … / sink = …directoryfeature
blackhole (sink)blackhole—
file, arrow (source + sink)file— (s3 / gcs / azure for an object store)
table (source)tablesqlite / postgres / mysql
nexmark (source)nexmark—
iceberg (sink)icebergiceberg
flight (sink)flightflight
kafka (source + sink) — the default when a dataset names neitherkafkakafka
nats (source + sink; JetStream)natsnats
stdout (sink) — the minimum connector, the worked examplestdout—
http (polling source), webhook (sink)httphttp
websocket (source: a live feed)websocketwebsocket

Re-exports§

pub use fv_streams_formats as formats;

Modules§

blackhole
sink = "blackhole": nowhere but the emitted counter — the sink of a measurement whose question is the engine, not the edge.
exactly_once
EXACTLY-ONCE DELIVERY for any sink, in one place. A connector supplies an EpochDestination — how one epoch’s rows reach its system, and (when that system can record it) which epoch reached it last — and sink returns the Sink that runs the protocol around it:
fanout
The reusable output fan-out — the core an Arrow-native output edge shares, independent of its transport. A BatchFanout is a Sink that strips the __fv_ meta columns and publishes each data batch to a broadcast channel; any number of readers subscribe and receive the same OutputMsg stream. The Flight output sink wraps one behind a gRPC server; another output edge can wrap one behind its own wire, so “serve the output as a live Arrow stream” lives in one place.
file
connector = "file" / sink = "file" (and "arrow", the short form of format = "arrow"): files and object stores at the edges — a source lists a directory or an s3:// prefix and reads its files straight into batches (no JSON decode); a sink writes the output as immutable parts of one fv_streams_formats format under a committed manifest, in two phases aligned with the checkpoint, so a crash never loses a committed row and a restored run continues where it committed. partitionBy writes Hive-style col=value/ directories; rollBytes rolls a part that grows past it; compression applies to Parquet.
messaging
MESSAGE HELPERS shared by every connector that reads or writes messages (Kafka, NATS, HTTP, WebSocket, stdout, and the next one): the message format, the drain that coalesces a poll into one batch, the decode of a drained run into one Arrow batch per partition, the delivery rule, and a row’s key on a broker. The message format itself (format = …: its decoders, its encoders and how a body holds many messages) lives in the contract crate, fv_streams_types::format, and is re-exported here. Public: a connector written outside this crate gets all of it too.
nexmark
connector = "nexmark": the deterministic Nexmark generator — in-process (inline = true) as a source whose tasks generate straight into Arrow batches, or written to the dataset’s Kafka topic by [spawn] so the Kafka source reads it (the topic mode; feature kafka).
reference
The connectors reference: docs/reference/connectors.md, rendered from each connector’s own description (crate::Connector::about) — the same description the dataset check reads, so the page and the check can never disagree. The runner renders it with every connector built in, and a test there fails when the file is stale (make reference rewrites it).
stdout
sink = "stdout": the output on standard output, a row per line in the dataset’s format (JSON lines by default, raw values or CSV records) — the smallest connector there is, and the worked example of what one needs: this file (the registration and the parse) and a sink. stderr = true writes to standard error instead (a runner’s own lines go to stdout).
table
connector = "table": a bounded database table (SQLite / Postgres / MySQL through sqlx, each a feature) as the lookup join’s side input — table or query (the database’s own SQL: pushdown by construction), refreshMs re-reads on schedule. Its size estimate sizes the join’s distribution.

Structs§

ConnectorDoc
What a connector is and every option it takes, per role — the one description the dataset check (check_dataset) and the generated reference (docs/reference/connectors.md, reference) both read.
Ctx
What a connector parses: the dataset’s name and its table from the pipeline definition, and the pipeline’s [brokers] table (Null when it has none): the default brokers, and the connection options, for a connector that reaches Kafka.
RoleDoc
One role of a connector (source or sink).

Constants§

RENAMED
Dataset options that were renamed, (old, new): an old name is still accepted wherever its new name is, with a warning that names the new one; setting both is refused.

Traits§

Connector
One connector: the connector = "…" spellings it reads as a source, the sink = "…" spellings it writes as a sink, and the parse of each from the dataset table into the seam’s kinds.

Functions§

check_dataset
Check a dataset’s keys against what its connectors take: every key must be connector, sink, or an option of the source the dataset names (or the default’s) or of its sink — or the old name of one (RENAMED). An unknown key is refused with the closest right one; the error lists what the dataset takes, the roles it names first. Ok holds a warning for each old name used. A dataset whose source or sink is not described is not checked.
connectors
Every connector this build carries.
resolve_sink
The sink a dataset’s sink = "…" names — or the default connector’s when it names none.
resolve_source
The source a dataset’s connector = "…" names — or the default connector’s when it names none (a build without the default names the gap). Err names the dataset and what is wrong.
settings
Every setting the connectors of this build read, by area: the shared source settings, then each connector’s, by title.