crabka_remote_storage_topic/
lib.rs

1//! Topic-backed [`RemoteLogMetadataManager`](crabka_remote_storage::RemoteLogMetadataManager) for Crabka, part of
2//! the KIP-405 tiered-storage stack.
3//!
4//! This crate ships [`TopicBasedRemoteLogMetadataManager`], the
5//! production replacement for [`crabka_remote_storage::InmemoryRemoteLogMetadataManager`].
6//! Remote-segment lifecycle events (add / update / partition-delete)
7//! are appended to an event log — in production the
8//! `__remote_log_metadata` Kafka topic — and every broker's local
9//! cache is rebuilt by consuming the same log. After a restart, a
10//! broker re-reads the topic from offset 0 and re-applies the full
11//! history to recover its cache.
12//!
13//! ## What this crate provides
14//!
15//! - [`TopicBasedRemoteLogMetadataManager`] — the
16//!   [`RemoteLogMetadataManager`](crabka_remote_storage::RemoteLogMetadataManager)
17//!   implementation.
18//! - [`MetadataEventLog`] — the publish/subscribe seam between the
19//!   manager and the underlying durable transport.
20//! - [`InProcessMetadataEventLog`] — an in-memory fixture for unit
21//!   tests and for modelling the multi-broker case without bringing
22//!   up a real cluster (multiple managers cloned from the same `Arc`
23//!   observe each other's writes).
24//! - [`MetadataEvent`] + the [`serde`] module — the on-wire binary
25//!   codec for the three event variants.
26//! - [`metadata_partition_for`] — the
27//!   `TopicIdPartition → metadata-topic-partition` hash.
28//! - [`KafkaMetadataEventLog`] — the production [`MetadataEventLog`]
29//!   adapter that wires the trait to
30//!   [`crabka_client_producer`] / [`crabka_client_core`] /
31//!   [`crabka_client_admin`], persisting events in the
32//!   `__remote_log_metadata` topic. Reads use manual per-partition
33//!   `Fetch` loops over `crabka_client_core` (no consumer group).
34//! - [`SwappableRlmm`] — the hot-swap facade the broker boots behind so
35//!   it can start on the fail-closed `NotReadyRlmm` and upgrade to the
36//!   topic-backed manager once its listener is serving.
37//!   `Broker::start` selects the topic-backed manager when the
38//!   `[remote_storage.kafka_metadata]` config section is present and
39//!   `in_memory` is not set to `true`.
40//!
41//! ## Operational boundaries
42//!
43//! The metadata topic is an append-only event log created with delete cleanup
44//! and infinite retention. The manager maintains a local snapshot cache so
45//! restarts can resume from committed per-partition offsets instead of replaying
46//! the full topic every time. It does not use a Kafka consumer group or broker
47//! offset commits; the broker drives assignments explicitly with
48//! [`TopicBasedRemoteLogMetadataManager::reconcile_assignment`]. Internal Kafka
49//! clients use plaintext loopback by default, or the TLS/SASL settings supplied
50//! through [`KafkaMetadataLogConfig::security`].
51//!
52//! ## In-process manager for tests and local tools
53//!
54//! ```no_run
55//! use crabka_remote_storage_topic::{
56//!     InProcessMetadataEventLog, TopicBasedRemoteLogMetadataManager,
57//! };
58//! use std::{path::PathBuf, time::Duration};
59//!
60//! # async fn run() -> Result<(), Box<dyn std::error::Error>> {
61//! let event_log = InProcessMetadataEventLog::new(16);
62//! let manager = TopicBasedRemoteLogMetadataManager::start(
63//!     event_log,
64//!     tokio::runtime::Handle::current(),
65//!     PathBuf::from("/var/lib/crabka/rlmm-cache"),
66//!     Duration::from_secs(30),
67//! )
68//! .await?;
69//!
70//! manager.reconcile_assignment(&[0, 1]).await;
71//! # Ok(())
72//! # }
73//! ```
74
75#![doc(html_root_url = "https://docs.rs/crabka-remote-storage-topic/0.3.5")]
76
77pub mod error;
78pub mod kafka_log;
79pub mod log;
80pub mod manager;
81pub mod not_ready;
82pub mod partitioning;
83pub mod serde;
84pub mod snapshot;
85pub mod swappable;
86
87pub use error::{CodecError, MetadataLogError, SnapshotError};
88pub use kafka_log::{
89    DEFAULT_NUM_PARTITIONS, DEFAULT_REPLICATION, KafkaMetadataEventLog, KafkaMetadataLogConfig,
90    METADATA_TOPIC,
91};
92pub use log::{
93    AssignmentHandle, InProcessMetadataEventLog, MetadataEventLog, MetadataEventRecord,
94    MetadataEventStream, PartitionStart,
95};
96pub use manager::TopicBasedRemoteLogMetadataManager;
97pub use not_ready::NotReadyRlmm;
98pub use partitioning::{metadata_partition_for, metadata_partitions_for};
99pub use serde::MetadataEvent;
100pub use snapshot::{SNAPSHOT_FILE_NAME, SNAPSHOT_FORMAT_VERSION, Snapshot};
101pub use swappable::SwappableRlmm;
crabka_remote_storage_topic/lib.rs

crabka_remote_storage_topic/
lib.rs
