Expand description
A buffered, multiplexing append log over pluggable object storage.
object-log stores an ordered, offset-addressed log as immutable objects in any
BlobStore (memory, local filesystem, or — behind a feature — S3). It deals
only in opaque payload bytes: it knows nothing about record formats, Kafka,
or brokers. Three pieces:
BlobStore— the storage port: durable-on-returnput,get,get_range,list,delete. AdaptersMemoryBlobStoreandLocalBlobStoreship in-crate.Sequencer— the sequencing seam: assigns offsets to durably-stored batches and owns the offset→location index.InMemorySequencerships in-crate; a consumer plugs its own (e.g. a Kafka coordinator) and the engine forwards each sequencer’sSequencer::Metauninterpreted.LogEngine— buffers and group-commits many batches into one object, PUTs it durably, then calls the sequencer, so PUT count is decoupled from produce count.LogEngine::produceresolves at a requestedDurability.
§Example
use object_log::{Durability, FlushConfig, InMemorySequencer, LogEngine, MemoryBlobStore, PartitionKey};
use bytes::Bytes;
use std::sync::Arc;
let engine = LogEngine::new(
Arc::new(MemoryBlobStore::new()),
Arc::new(InMemorySequencer::new()),
FlushConfig::default(),
"log/",
);
let p = PartitionKey("events-0".into());
let out = engine
.produce(p.clone(), Bytes::from_static(b"hello"), 1, (), Durability::Sequenced)
.await
.unwrap();
assert_eq!(out.base_offset, Some(0));
let batches = engine.fetch(&p, 0, 1024).await.unwrap();
assert_eq!(batches[0].payload, "hello");Structs§
- Append
Outcome - Outcome of a
LogEngine::producecall. - Batch
Location - Where a batch’s bytes live inside its object. Authored by the engine (it
owns object layout); the sequencer stores it in the index and returns it from
Sequencer::lookup. - Budget
Config - Configuration for the media-ops budget controller.
- Buffer
Stats - Snapshot of the engine’s buffering envelope.
- Commit
Batch - One batch presented to
Sequencer::commit. The engine fillspartition,record_count, andlocation(all engine-visible);metais forwarded uninterpreted — only the sequencer reads it. - Effective
Knob - One inspectable knob with layered sources.
- Fetched
Batch - A batch read back by
LogEngine::fetch, with its assigned base offset. - Flush
Config - Flush-trigger policy for the engine’s group-commit buffer.
- InMemory
Sequencer - A single-node, in-memory
Sequencer(Meta = ()): assigns dense offsets and keeps the offset→location index in memory. - Index
Entry - An index entry resolving an offset range to its bytes.
- Index
Snapshot - Full index view from
ManifestSequencer::snapshot. - Local
Blob Store - Filesystem-backed
BlobStorerooted at a directory. - LogEngine
- A buffered, multiplexing append-log engine over a
BlobStore, with sequencing delegated to aSequencer. - Manifest
Sequencer - A
Sequencer(Meta = ()) that persists its index to aBlobStore. - Media
OpStats - Durable-media accounting for the flush budget controller (TD-004).
- Memory
Blob Store - In-process
BlobStorebacked by a map. For tests and development; state is lost on drop and is not crash-durable. - Partition
Key - Opaque, engine-visible identifier for an independent offset stream (one dense,
monotonic offset space). A Kafka broker maps
(topic, partition)onto one of these; a WAL maps a shard. object-log treats it as an opaque key. - Partition
Snapshot - One partition’s index bounds and entries.
- Pipeline
Snapshot - Point-in-time controller + buffer view.
Enums§
- Budget
Mode - How the engine resolves budget pressure vs admitted latency.
- Commit
Outcome - Per-batch result of
Sequencer::commit. - Durability
- The durability point a
LogEngine::producecall resolves at. - Effective
Reason - Why an effective value was chosen (for inspectability).
- Object
LogError - Errors returned by the object-log engine, storage adapters, and sequencer.
Traits§
- Blob
Store - A minimal async object store over immutable, string-keyed blobs.
- Sequencer
- The linearization point: assigns offsets to durably-stored batches and owns the offset→location index.
Functions§
- reap_
orphans - Delete objects under
data_prefixwhose keys are absent fromlive.