Crate sources_core

Expand description

§flusso-sources-core

The source abstractions for flusso — split into two independent halves, what changed? and what to build?

§Quick reference

Item	Role
`cdc`	What changed? — pluggable change capture: a stream of `Change`s, progress confirmed via an `Ack`
`document`	What to build? — turns a changed row (table + key) into the documents it affects, and assembles each
`validate_indexes`	Source-independent half of the schema/DB type check; a source supplies only a `Catalog` over its column types
`CaptureProvisioning`	Reports whether a source can stream an index’s tables and — when privileged — provisions the gap
`RowKey`	A row’s primary key as ordered column/value pairs
`SnapshotTable`	A schema-qualified table to snapshot when seeding an index
`SourceSpec`	The source’s own view of what to build — the enabled indexes + their schemas, a subset of the top-level config
`SourceError` / `Result`	The common error type

The two responsibilities each get their own module. Neither references the other; the engine is the only thing that bridges them. Mix any change mechanism with any document builder, and test or replace either without touching the other.

cdc yields thin Changes and confirms progress via an Ack. Logical replication (WAL) is the first mechanism; polling or triggers can follow.

validate_indexes leans on the fact that a self-describing schema states its own types, so the mapping derives without a database. A reachable database is used only to confirm the declared types and nullability match the real columns.

CaptureProvisioning is mechanism-neutral by design — the daemon and CLI drive it without ever naming “publication” (Postgres’s gap-filling mechanism). Given the tables an index reads (SourceSpec::all_tables), it reports coverage and provisions what’s missing.

ℹ️ Info — SourceSpec is the composition root’s gift to the backend: the source crate never sees the assembled Config. It gets only the enabled indexes and their schemas, in schema-core types.

Modules§

cdc: Change data capture: what changed?
document: Document construction: what to build?

Structs§

ColumnInfo: How a store reports one base-table column: its native type name (as the store spells it, e.g. Postgres character varying(255)) and whether it admits null.
CoverageReport: What a source found when asked whether it can stream a set of tables.
Diagnostic: One disagreement between a declared schema and the live database.
QualifiedTable: A schema-qualified table, the unit a source needs to reason about coverage (which tables it must be able to stream). Ordered by (schema, table) so a BTreeSet of them is deterministic.
RowKey: A row’s primary key, as ordered column/value pairs.
SnapshotTable: A schema-qualified table to snapshot during an initial backfill.
SourceSpec: The enabled indexes a source must build, each paired with its schema.

Enums§

Severity: How serious a Diagnostic is.
SourceError: Why a ChangeCapture failed to start or to produce the next change.

Traits§

CaptureProvisioning: A source’s ability to report and provision the prerequisites for streaming a set of tables. Implemented per mechanism (Postgres backs it with a publication); consumed by the CLI (check reports, run ensures) only through this neutral surface.
Catalog: A source’s view of its own catalog: the type and nullability of a base-table column. This is all validate_indexes needs from a source to check a declared schema against the live store.

Functions§

validate_indexes: Validate every index in spec against the store behind catalog, returning the disagreements found. An empty result means the declared schema matches the database. The spec already holds only enabled indexes.

Type Aliases§

Result: Result alias for source operations.