Expand description
A buffered, multiplexing append log over pluggable object storage.
object-log stores an ordered, offset-addressed log as immutable objects in any
BlobStore (memory, local filesystem, or — behind a feature — S3). It deals
only in opaque payload bytes: it knows nothing about record formats, Kafka,
or brokers. Three pieces:
BlobStore— the storage port: durable-on-returnput,get,get_range,list,delete. AdaptersMemoryBlobStoreandLocalBlobStoreship in-crate.Sequencer— the sequencing seam: assigns offsets to durably-stored batches and owns the offset→location index.InMemorySequencerships in-crate; a consumer plugs its own (e.g. a Kafka coordinator) and the engine forwards each sequencer’sSequencer::Metauninterpreted.LogEngine— buffers and group-commits many batches into one object, PUTs it durably, then calls the sequencer, so PUT count is decoupled from produce count.LogEngine::produceresolves at a requestedDurability.
§Example
use object_log::{Durability, FlushConfig, InMemorySequencer, LogEngine, MemoryBlobStore, PartitionKey};
use bytes::Bytes;
use std::sync::Arc;
let engine = LogEngine::new(
Arc::new(MemoryBlobStore::new()),
Arc::new(InMemorySequencer::new()),
FlushConfig::default(),
"log/",
);
let p = PartitionKey("events-0".into());
let out = engine
.produce(p.clone(), Bytes::from_static(b"hello"), 1, (), Durability::Sequenced)
.await
.unwrap();
assert_eq!(out.base_offset, Some(0));
let batches = engine.fetch(&p, 0, 1024).await.unwrap();
assert_eq!(batches[0].payload, "hello");Structs§
- Append
Outcome - Outcome of a
LogEngine::producecall. - Authoritative
Partition State - Point-in-time durable bounds; grants no exclusivity or pinned history view.
- Batch
Location - Where a batch’s bytes live inside its object. Authored by the engine (it
owns object layout); the sequencer stores it in the index and returns it from
Sequencer::lookup. - Budget
Config - Configuration for the media-ops budget controller.
- Buffer
Stats - Snapshot of the engine’s buffering envelope.
- Commit
Batch - One batch presented to
Sequencer::commit. The engine fillspartition,record_count, andlocation(all engine-visible);metais forwarded uninterpreted — only the sequencer reads it. - Effective
Knob - One inspectable knob with layered sources.
- Fetch
Plan - Unforgeable metadata page from one sequencer/blob authority. Reserve the declared metadata budget before constructing, then per-batch bytes before read.
- Fetched
Batch - A batch read back by
LogEngine::fetch, with its assigned base offset. - Flush
Config - Flush-trigger policy for the engine’s group-commit buffer.
- InMemory
Sequencer - A single-node, in-memory
Sequencer(Meta = ()): assigns dense offsets and keeps the offset→location index in memory. - Index
Entry - An index entry resolving an offset range to its bytes.
- Index
Snapshot - Full index view from
ManifestSequencer::snapshot. - Local
Blob Store - Filesystem-backed
BlobStorerooted at a directory. - LogEngine
- A buffered, multiplexing append-log engine over a
BlobStore, with sequencing delegated to aSequencer. - Manifest
Sequencer - A
Sequencer(Meta = ()) that persists its index to aBlobStore. - Media
OpStats - Durable-media accounting for the flush budget controller (TD-004).
- Memory
Blob Store - In-process
BlobStorebacked by a map. For tests and development; state is lost on drop and is not crash-durable. - Metadata
Limits - Explicit source-memory limits for both reads and conditional writes. Transport/header-parser allocations require separate qualification.
- Metadata
Owners - Separate ownership of in-flight physical work and retained observation bytes.
Returned snapshots retain only
retained; upload/request sources retain both. - Metadata
Snapshot - An opaque observation binding one adapter instance, literal key and version. Payload ownership remains with the snapshot; callers reserve payload, key, token and transport scratch before invoking the physical operation.
- Partition
Key - Opaque, engine-visible identifier for an independent offset stream (one dense,
monotonic offset space). A Kafka broker maps
(topic, partition)onto one of these; a WAL maps a shard. object-log treats it as an opaque key. - Partition
Snapshot - One partition’s index bounds and entries.
- Pipeline
Snapshot - Point-in-time controller + buffer view.
Enums§
- Budget
Mode - How the engine resolves budget pressure vs admitted latency.
- CasOutcome
- Result of
BlobStore::compare_and_swap. - Commit
Outcome - Per-batch result of
Sequencer::commit. - Conditional
Commit Outcome - Opt-in conditional results; existing CommitOutcome encoding stays unchanged.
- Durability
- The durability point a
LogEngine::producecall resolves at. - Effective
Reason - Why an effective value was chosen (for inspectability).
- Metadata
Swap - A definite metadata conditional-write outcome. Transport errors can follow durable publication and must never be interpreted as definite rejection.
- Object
LogError - Errors returned by the object-log engine, storage adapters, and sequencer.
Traits§
- Blob
Store - A minimal async object store over immutable, string-keyed blobs.
- Bounded
Metadata Store - Required bounded authority operations, independent of legacy BlobStore.
There are deliberately no default implementations.
limitsbounds every retained payload and token, including conflict observations; transport/header and operation ownership require separate caller admission. Cancellation does not prove a conditional write had no effects. Owners must drive and drain it. Input/output source buffers retain the supplied lifetime owner. This does not attest to all SDK transport allocations or provider-side effects after an ambiguous request; those require separate transport/durability qualification. - Bounded
Sequencer - Compile-time native page port; legacy custom sequencers need not implement it.
- Sequencer
- The linearization point: assigns offsets to durably-stored batches and owns the offset→location index.
Functions§
- reap_
orphans - Delete objects under
data_prefixwhose keys are absent fromlive.
Type Aliases§
- Append
Owner - An embedding operation lifetime, retained through queued work and started upload completion, including copied upload buffers and failure draining. It must not depend on the caller waiting for its acknowledgement.