Crate crabka_log 

Byte-compatible reader/writer for Apache Kafka’s on-disk log format.

This crate provides the append-only storage layer used by the Crabka broker. It reads and writes Kafka 4.x’s on-disk log format byte-for-byte: 20-digit zero-padded segment filenames, sparse .index and .timeindex files, append-only .log files containing crabka_protocol::records::RecordBatch v2 streams.

What this crate does

• Open + recover existing log directories.
• Append RecordBatches to the active segment.
• Read sequentially from an absolute offset.
• Truncate the log to an offset (for replication / leader election).
• Time-based and size-based retention.

Scope and boundaries

This crate is the byte-compatible segment/index layer. It exposes append, read, truncation, leader-epoch checkpoint, transaction-index, retention, and compaction primitives over a single log directory. Broker-level policy — topic configuration, leader/follower ownership, tiered-storage scheduling, transaction visibility rules, and write serialization — is applied by crabka-broker above this storage layer.

Quick start

use crabka_log::{Log, LogConfig};
use crabka_protocol::records::RecordBatch;

let mut log = Log::open("/var/kafka/my-topic-0", LogConfig::default()).unwrap();
let mut batch = RecordBatch::default();
// ... fill the batch ...
let assigned_offset = log.append(&mut batch).unwrap();

let out = log.read(0, 1024 * 1024).unwrap();

Exporting a segment

use crabka_log::{Log, LogConfig};

let mut log = Log::open("/var/kafka/my-topic-0", LogConfig::default())?;
for segment in log.tierable_segments() {
    println!(
        "segment {} starts at offset {}",
        segment.log_path.display(),
        segment.base_offset
    );
}

Structs

AbortedTxn

EpochEntry

LeaderEpochCheckpoint

Log

A Kafka-format log: a sorted collection of Segments plus a single active segment that accepts appends.

LogConfig

Tunables for Log behavior.

RawRead

Verbatim, decode-free output of Log::read_raw.

RawSegmentRead

Verbatim, decode-free output of Segment::read_raw.

ReadOutput

Result of Log::read: the absolute offset of the first batch returned and the batches themselves.

Segment

A single log segment: the .log data file paired with its sparse .index (offset → byte position) and .timeindex (timestamp → relative offset) sidecars.

SegmentExport

A sealed segment described for tiered-storage offload (KIP-405). Carries the on-disk file paths plus the offset / timestamp / size metadata and the leader-epoch ranges a RemoteLogManager needs to build remote-segment metadata. Produced by Log::tierable_segments.

TxnIndex

Enums

CleanupPolicy

Per-topic policy for what to do with old log segments.

LogError

Errors returned by Log and Segment.