Crate crabka_connect 

Connector-framework SPI for Crabka.

This crate defines the keystone traits that every connector — every CDC source, every telemetry sink — builds on, plus the converter layer that bridges typed payloads to the Kafka wire.

• Source pulls records out of an external system one at a time (poll), snapshots its read position (checkpoint), and restores it on restart (seek). Sources can optionally acknowledge a persisted checkpoint after sink commit with acknowledge.

• Sink pushes records into an external system in batches (put), makes them durable (flush), and optionally gates writes behind a transaction for exactly-once delivery (begin / commit / abort).

• Converter bridges a connector’s typed payload T to and from wire bytes::Bytes. ByteIdentity is a byte-for-byte passthrough; SchemaConverter wraps the Confluent schema-registry serdes from crabka-schema-serde.

• ConnectorRuntime is the embeddable, single-process driver that owns a Source + Sink pair and pipes records between them — polling, applying bounded backpressure, coordinating source checkpoints with sink commits through the transactional gate, and exposing pause/resume + graceful-drain lifecycle hooks. No Connect worker protocol, no REST.

The transport traits mirror the streams runtime I/O traits (RecordFetcher / RecordProducer) and the remote-storage RemoteStorageManager SPI; the converter layer mirrors Kafka Connect’s Converter. All records carry an optional key, optional value (a None value is a tombstone), an optional timestamp, and ordered headers — see ConnectRecord.

Re-exports

pub use config::ConfigDef;

pub use config::ConfigError;

pub use config::ConfigKey;

pub use config::ConfigKind;

pub use config::ConfigResult;

pub use config::ConnectorConfig;

pub use config::EnvSecretResolver;

pub use config::FromResolvedValue;

pub use config::RawConfig;

pub use config::ResolveOptions;

pub use config::ResolvedConfig;

pub use config::SecretRef;

pub use config::SecretResolutionError;

pub use config::SecretResolver;

pub use config::SecretString;

pub use convert::ByteIdentity;

pub use convert::Converter;

pub use convert::SchemaConverter;

pub use error::ConnectError;

pub use ids::PartitionMap;

pub use ids::PositionMap;

pub use record::ConnectRecord;

pub use record::Header;

pub use record::OffsetMap;

pub use record::OffsetValue;

pub use record::SourceOffset;

pub use runtime::CheckpointStore;

pub use runtime::ConnectorHandle;

pub use runtime::ConnectorRuntime;

pub use runtime::HasSink;

pub use runtime::HasSource;

pub use runtime::InMemoryCheckpointStore;

pub use runtime::NoSink;

pub use runtime::NoSource;

pub use runtime::RuntimeState;

pub use sink::Sink;

pub use source::Source;

Modules

config

Connector configuration definitions, validation, and secret resolution.

convert

The converter layer: bridge a connector’s typed payload T to and from the raw Bytes that travel on the Kafka wire.

error

The error type shared by the connector SPI.

ids

Newtypes over the connector-offset maps.

record

The record and offset types that flow across the connector SPI.

runtime

The ConnectorRuntime: an async driver that owns one Source and one Sink and pipes records between them in a single process — no Connect worker protocol, no REST, no broker required between the two ends. This is the embeddable, single-binary shape (a ground-station / aircraft edge box): build it programmatically, run it, and drive its lifecycle through the returned ConnectorHandle.

sink

The Sink SPI: push records into an external system, with an optional transactional write gate for exactly-once delivery.

source

The Source SPI: pull records out of an external system, with a checkpointable read position.

Derive Macros

ConnectorConfig