1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
//! Topic-backed [`RemoteLogMetadataManager`](crabka_remote_storage::RemoteLogMetadataManager) for Crabka, part of
//! the KIP-405 tiered-storage stack.
//!
//! This crate ships [`TopicBasedRemoteLogMetadataManager`], the
//! production replacement for [`crabka_remote_storage::InmemoryRemoteLogMetadataManager`].
//! Remote-segment lifecycle events (add / update / partition-delete)
//! are appended to an event log — in production the
//! `__remote_log_metadata` Kafka topic — and every broker's local
//! cache is rebuilt by consuming the same log. After a restart, a
//! broker re-reads the topic from offset 0 and re-applies the full
//! history to recover its cache.
//!
//! ## What this crate provides
//!
//! - [`TopicBasedRemoteLogMetadataManager`] — the
//! [`RemoteLogMetadataManager`](crabka_remote_storage::RemoteLogMetadataManager)
//! implementation.
//! - [`MetadataEventLog`] — the publish/subscribe seam between the
//! manager and the underlying durable transport.
//! - [`InProcessMetadataEventLog`] — an in-memory fixture for unit
//! tests and for modelling the multi-broker case without bringing
//! up a real cluster (multiple managers cloned from the same `Arc`
//! observe each other's writes).
//! - [`MetadataEvent`] + the [`serde`] module — the on-wire binary
//! codec for the three event variants.
//! - [`metadata_partition_for`] — the
//! `TopicIdPartition → metadata-topic-partition` hash.
//! - [`KafkaMetadataEventLog`] — the production [`MetadataEventLog`]
//! adapter that wires the trait to
//! [`crabka_client_producer`] / [`crabka_client_core`] /
//! [`crabka_client_admin`], persisting events in the
//! `__remote_log_metadata` topic. Reads use manual per-partition
//! `Fetch` loops over `crabka_client_core` (no consumer group).
//! - [`SwappableRlmm`] — the hot-swap facade the broker boots behind so
//! it can start on the fail-closed `NotReadyRlmm` and upgrade to the
//! topic-backed manager once its listener is serving.
//! `Broker::start` selects the topic-backed manager when the
//! `[remote_storage.kafka_metadata]` config section is present and
//! `in_memory` is not set to `true`.
//!
//! ## Operational boundaries
//!
//! The metadata topic is an append-only event log created with delete cleanup
//! and infinite retention. The manager maintains a local snapshot cache so
//! restarts can resume from committed per-partition offsets instead of replaying
//! the full topic every time. It does not use a Kafka consumer group or broker
//! offset commits; the broker drives assignments explicitly with
//! [`TopicBasedRemoteLogMetadataManager::reconcile_assignment`]. Internal Kafka
//! clients use plaintext loopback by default, or the TLS/SASL settings supplied
//! through [`KafkaMetadataLogConfig::security`].
//!
//! ## In-process manager for tests and local tools
//!
//! ```no_run
//! use crabka_remote_storage_topic::{
//! InProcessMetadataEventLog, TopicBasedRemoteLogMetadataManager,
//! };
//! use std::{path::PathBuf, time::Duration};
//!
//! # async fn run() -> Result<(), Box<dyn std::error::Error>> {
//! let event_log = InProcessMetadataEventLog::new(16);
//! let manager = TopicBasedRemoteLogMetadataManager::start(
//! event_log,
//! tokio::runtime::Handle::current(),
//! PathBuf::from("/var/lib/crabka/rlmm-cache"),
//! Duration::from_secs(30),
//! )
//! .await?;
//!
//! manager.reconcile_assignment(&[0, 1]).await;
//! # Ok(())
//! # }
//! ```
pub use ;
pub use ;
pub use ;
pub use TopicBasedRemoteLogMetadataManager;
pub use NotReadyRlmm;
pub use ;
pub use MetadataEvent;
pub use ;
pub use SwappableRlmm;