Crate kimberlite_storage 

kmb-storage: Append-only segment storage for Kimberlite

This crate implements the durable event log storage layer. Events are stored in segment files with a simple binary format that includes cryptographic hash chains for tamper detection and CRC32 checksums for corruption detection.

Record Format

Each record is stored as:

[offset:i64][prev_hash:32B][length:u32][payload:bytes][crc32:u32]
    8B           32B           4B         variable        4B

• offset: The logical position of this event in the stream
• prev_hash: SHA-256 hash of the previous record (all zeros for genesis)
• length: Size of the payload in bytes
• payload: The event data
• crc32: Checksum of all preceding fields for corruption detection

Hash Chain

Records form a tamper-evident chain where each record includes the hash of the previous record. This allows verification that the log has not been modified:

Record 0: prev_hash = [0; 32]  →  hash_0 = SHA-256(payload_0)
Record 1: prev_hash = hash_0   →  hash_1 = SHA-256(hash_0 || payload_1)
Record 2: prev_hash = hash_1   →  hash_2 = SHA-256(hash_1 || payload_2)

File Layout

data_dir/
  {stream_id}/
    segment_000000.log   # First segment (future: rotation)
    segment_000001.log   # Second segment, etc.

Example

use kimberlite_storage::Storage;
use kimberlite_types::{Offset, StreamId};
use bytes::Bytes;

let storage = Storage::new("/data/kimberlite");

// Append events
let events = vec![Bytes::from("event1"), Bytes::from("event2")];
let new_offset = storage.append_batch(
    StreamId::new(1),
    events,
    Offset::new(0),
    true,  // fsync for durability
)?;

// Read events back
let events = storage.read_from(StreamId::new(1), Offset::new(0), 1024)?;

Re-exports

pub use codec::Codec;

pub use codec::CodecRegistry;

pub use codec::Lz4Codec;

pub use codec::NoneCodec;

pub use codec::ZstdCodec;

Modules

codec

Compression codecs for record payloads.

Structs

AppendPipeline

Two-stage append pipeline that overlaps CPU preparation with I/O.

CheckpointIndex

In-memory sparse index of checkpoint offsets.

CompactionConfig

Configuration for log compaction.

CompactionResult

Result of a compaction operation.

MemoryStorage

Pure in-memory storage backend.

OffsetIndex

Maps logical offset → physical byte position for O(1) lookups.

PreparedBatch

A batch of records that has been serialized and is ready for I/O.

Record

A single record in the event log.

Storage

Append-only event log storage with checkpoint support and segment rotation.

Enums

StorageError

Errors that can occur during storage operations.

Traits

StorageBackend

Abstract storage backend used by KimberliteInner’s effect executor.

Functions

create_checkpoint

Creates a Checkpoint from deserialized checkpoint data.

deserialize_checkpoint_payload

Deserializes checkpoint data from a record payload.

serialize_checkpoint_payload

Serializes checkpoint data to bytes for storage as a record payload.

should_create_checkpoint

Determines if a checkpoint should be created at the given offset.