Skip to main content

Crate object_log

Crate object_log 

Source
Expand description

A buffered, multiplexing append log over pluggable object storage.

object-log stores an ordered, offset-addressed log as immutable objects in any BlobStore (memory, local filesystem, or — behind a feature — S3). It deals only in opaque payload bytes: it knows nothing about record formats, Kafka, or brokers. Three pieces:

  • BlobStore — the storage port: durable-on-return put, get, get_range, list, delete. Adapters MemoryBlobStore and LocalBlobStore ship in-crate.
  • Sequencer — the sequencing seam: assigns offsets to durably-stored batches and owns the offset→location index. InMemorySequencer ships in-crate; a consumer plugs its own (e.g. a Kafka coordinator) and the engine forwards each sequencer’s Sequencer::Meta uninterpreted.
  • LogEngine — buffers and group-commits many batches into one object, PUTs it durably, then calls the sequencer, so PUT count is decoupled from produce count. LogEngine::produce resolves at a requested Durability.

§Example

use object_log::{Durability, FlushConfig, InMemorySequencer, LogEngine, MemoryBlobStore, PartitionKey};
use bytes::Bytes;
use std::sync::Arc;

let engine = LogEngine::new(
    Arc::new(MemoryBlobStore::new()),
    Arc::new(InMemorySequencer::new()),
    FlushConfig::default(),
    "log/",
);
let p = PartitionKey("events-0".into());

let out = engine
    .produce(p.clone(), Bytes::from_static(b"hello"), 1, (), Durability::Sequenced)
    .await
    .unwrap();
assert_eq!(out.base_offset, Some(0));

let batches = engine.fetch(&p, 0, 1024).await.unwrap();
assert_eq!(batches[0].payload, "hello");

Structs§

AppendOutcome
Outcome of a LogEngine::produce call.
BatchLocation
Where a batch’s bytes live inside its object. Authored by the engine (it owns object layout); the sequencer stores it in the index and returns it from Sequencer::lookup.
BudgetConfig
Configuration for the media-ops budget controller.
BufferStats
Snapshot of the engine’s buffering envelope.
CommitBatch
One batch presented to Sequencer::commit. The engine fills partition, record_count, and location (all engine-visible); meta is forwarded uninterpreted — only the sequencer reads it.
EffectiveKnob
One inspectable knob with layered sources.
FetchedBatch
A batch read back by LogEngine::fetch, with its assigned base offset.
FlushConfig
Flush-trigger policy for the engine’s group-commit buffer.
InMemorySequencer
A single-node, in-memory Sequencer (Meta = ()): assigns dense offsets and keeps the offset→location index in memory.
IndexEntry
An index entry resolving an offset range to its bytes.
IndexSnapshot
Full index view from ManifestSequencer::snapshot.
LocalBlobStore
Filesystem-backed BlobStore rooted at a directory.
LogEngine
A buffered, multiplexing append-log engine over a BlobStore, with sequencing delegated to a Sequencer.
ManifestSequencer
A Sequencer (Meta = ()) that persists its index to a BlobStore.
MediaOpStats
Durable-media accounting for the flush budget controller (TD-004).
MemoryBlobStore
In-process BlobStore backed by a map. For tests and development; state is lost on drop and is not crash-durable.
PartitionKey
Opaque, engine-visible identifier for an independent offset stream (one dense, monotonic offset space). A Kafka broker maps (topic, partition) onto one of these; a WAL maps a shard. object-log treats it as an opaque key.
PartitionSnapshot
One partition’s index bounds and entries.
PipelineSnapshot
Point-in-time controller + buffer view.

Enums§

BudgetMode
How the engine resolves budget pressure vs admitted latency.
CasOutcome
Result of BlobStore::compare_and_swap.
CommitOutcome
Per-batch result of Sequencer::commit.
Durability
The durability point a LogEngine::produce call resolves at.
EffectiveReason
Why an effective value was chosen (for inspectability).
ObjectLogError
Errors returned by the object-log engine, storage adapters, and sequencer.

Traits§

BlobStore
A minimal async object store over immutable, string-keyed blobs.
Sequencer
The linearization point: assigns offsets to durably-stored batches and owns the offset→location index.

Functions§

reap_orphans
Delete objects under data_prefix whose keys are absent from live.