Crate crabka_remote_storage_topic 

Topic-backed RemoteLogMetadataManager for Crabka, part of the KIP-405 tiered-storage stack.

This crate ships TopicBasedRemoteLogMetadataManager, the production replacement for crabka_remote_storage::InmemoryRemoteLogMetadataManager. Remote-segment lifecycle events (add / update / partition-delete) are appended to an event log — in production the __remote_log_metadata Kafka topic — and every broker’s local cache is rebuilt by consuming the same log. After a restart, a broker re-reads the topic from offset 0 and re-applies the full history to recover its cache.

What this crate provides

• TopicBasedRemoteLogMetadataManager — the RemoteLogMetadataManager implementation.
• MetadataEventLog — the publish/subscribe seam between the manager and the underlying durable transport.
• InProcessMetadataEventLog — an in-memory fixture for unit tests and for modelling the multi-broker case without bringing up a real cluster (multiple managers cloned from the same Arc observe each other’s writes).
• MetadataEvent + the serde module — the on-wire binary codec for the three event variants.
• metadata_partition_for — the TopicIdPartition → metadata-topic-partition hash.
• KafkaMetadataEventLog — the production MetadataEventLog adapter that wires the trait to crabka_client_producer / crabka_client_core / crabka_client_admin, persisting events in the __remote_log_metadata topic. Reads use manual per-partition Fetch loops over crabka_client_core (no consumer group).
• SwappableRlmm — the hot-swap facade the broker boots behind so it can start on the fail-closed NotReadyRlmm and upgrade to the topic-backed manager once its listener is serving. Broker::start selects the topic-backed manager when the [remote_storage.kafka_metadata] config section is present and in_memory is not set to true.

Operational boundaries

The metadata topic is an append-only event log created with delete cleanup and infinite retention. The manager maintains a local snapshot cache so restarts can resume from committed per-partition offsets instead of replaying the full topic every time. It does not use a Kafka consumer group or broker offset commits; the broker drives assignments explicitly with TopicBasedRemoteLogMetadataManager::reconcile_assignment. Internal Kafka clients use plaintext loopback by default, or the TLS/SASL settings supplied through KafkaMetadataLogConfig::security.

In-process manager for tests and local tools

use crabka_remote_storage_topic::{
    InProcessMetadataEventLog, TopicBasedRemoteLogMetadataManager,
};
use std::{path::PathBuf, time::Duration};

let event_log = InProcessMetadataEventLog::new(16);
let manager = TopicBasedRemoteLogMetadataManager::start(
    event_log,
    tokio::runtime::Handle::current(),
    PathBuf::from("/var/lib/crabka/rlmm-cache"),
    Duration::from_secs(30),
)
.await?;

manager.reconcile_assignment(&[0, 1]).await;

Re-exports

pub use error::CodecError;

pub use error::MetadataLogError;

pub use error::SnapshotError;

pub use kafka_log::DEFAULT_NUM_PARTITIONS;

pub use kafka_log::DEFAULT_REPLICATION;

pub use kafka_log::KafkaMetadataEventLog;

pub use kafka_log::KafkaMetadataLogConfig;

pub use kafka_log::METADATA_TOPIC;

pub use log::AssignmentHandle;

pub use log::InProcessMetadataEventLog;

pub use log::MetadataEventLog;

pub use log::MetadataEventRecord;

pub use log::MetadataEventStream;

pub use log::PartitionStart;

pub use manager::TopicBasedRemoteLogMetadataManager;

pub use not_ready::NotReadyRlmm;

pub use partitioning::metadata_partition_for;

pub use partitioning::metadata_partitions_for;

pub use serde::MetadataEvent;

pub use snapshot::SNAPSHOT_FILE_NAME;

pub use snapshot::SNAPSHOT_FORMAT_VERSION;

pub use snapshot::Snapshot;

pub use swappable::SwappableRlmm;

Modules

error

Errors surfaced by the topic-backed RemoteLogMetadataManager and the underlying MetadataEventLog.

kafka_log

KafkaMetadataEventLog — the production MetadataEventLog adapter that persists events in the internal __remote_log_metadata Kafka topic.

log

MetadataEventLog: the publish/subscribe seam between the TopicBasedRemoteLogMetadataManager and the underlying durable event store.

manager

TopicBasedRemoteLogMetadataManager — production RemoteLogMetadataManager implementation backed by a publish / subscribe MetadataEventLog.

not_ready

NotReadyRlmm — fail-closed RemoteLogMetadataManager placeholder.

partitioning

Deterministic TopicIdPartition → metadata-topic-partition mapping.

serde

Binary wire format for __remote_log_metadata events.

snapshot

On-disk RLMM snapshot: a versioned envelope wrapping a RlmmCacheDump plus the per-metadata-partition committed offsets, so a restarting broker resumes the metadata consumer from committed + 1 instead of replaying __remote_log_metadata from offset 0.

swappable

SwappableRlmm — a RemoteLogMetadataManager facade whose backing implementation can be replaced atomically after construction.