Crate spate_core 

The engine of the Spate framework.

spate-core contains the pipeline runtime and every technology-neutral abstraction: records and their checkpoint tokens, the operator chain, the source and sink traits, checkpointing, backpressure, configuration loading, metrics, and the admin server.

Applications should depend on the spate facade crate rather than on spate-core directly.

The architecture and its invariants are documented in docs/DESIGN.md; the metric taxonomy in docs/METRICS.md.

Re-exports

pub use error::DeserError;

pub use error::ErrorClass;

pub use error::ErrorPolicy;

pub use error::FatalError;

pub use error::SinkError;

pub use error::SourceError;

pub use framing::FramingContract;

pub use record::Flow;

pub use record::PartitionId;

pub use record::RawPayload;

pub use record::Record;

pub use record::RecordMeta;

pub use bytes;

Modules

admin

Admin HTTP server: /metrics, /healthz, and /readyz.

backpressure

Backpressure: the global in-flight byte budget and the watermark pause/resume controller with hysteresis.

checkpoint

Checkpointing: acknowledgement tracking and watermark commits.

config

Pipeline configuration: typed framework sections plus opaque per-component passthrough, loaded from YAML with ${VAR:-default} environment interpolation.

coordination

Split coordination: the seam that lets several pipeline instances divide one broker-less source’s work without duplicates.

deser

Deserialization contract: one borrowed payload in, 0..N records out, push-style.

error

Error taxonomy and per-stage error policies.

framing

Streaming record framing: the seam that cuts a decoded byte stream into records.

metrics

Metrics: exporter installation and pre-registered handle structs for every pipeline stage.

ops

Operator chain: statically composed push stages behind one type-erasure boundary per batch.

pipeline

Pipeline runtime: pinned driver threads, the source controller, and process assembly.

record

Core record types: payloads, metadata, and the flow-control signal.

sink

Sink abstraction: pipeline threads encode, shard workers batch and write.

source

Source abstraction: a control plane (Source) and a data plane (SourceLane).

telemetry

Structured logging: tracing initialisation (JSON for Kubernetes) and rate-limited hot-path logging helpers.

Macros

rate_limited_warn

tracing::warn! behind a RateLimit. When events were suppressed since the last allowed one, the emitted line carries a suppressed field with the count.