Skip to main content

Crate object_log

Crate object_log 

Source
Expand description

A buffered, multiplexing append log over pluggable object storage.

object-log stores an ordered, offset-addressed log as immutable objects in any BlobStore (memory, local filesystem, or — behind a feature — S3). It deals only in opaque payload bytes: it knows nothing about record formats, Kafka, or brokers. Three pieces:

  • BlobStore — the storage port: durable-on-return put, get, get_range, list, delete. Adapters MemoryBlobStore and LocalBlobStore ship in-crate.
  • Sequencer — the sequencing seam: assigns offsets to durably-stored batches and owns the offset→location index. InMemorySequencer ships in-crate; a consumer plugs its own (e.g. a Kafka coordinator) and the engine forwards each sequencer’s Sequencer::Meta uninterpreted.
  • LogEngine — buffers and group-commits many batches into one object, PUTs it durably, then calls the sequencer, so PUT count is decoupled from produce count. LogEngine::produce resolves at a requested Durability.

§Example

use object_log::{Durability, FlushConfig, InMemorySequencer, LogEngine, MemoryBlobStore, PartitionKey};
use bytes::Bytes;
use std::sync::Arc;

let engine = LogEngine::new(
    Arc::new(MemoryBlobStore::new()),
    Arc::new(InMemorySequencer::new()),
    FlushConfig::default(),
    "log/",
);
let p = PartitionKey("events-0".into());

let out = engine
    .produce(p.clone(), Bytes::from_static(b"hello"), 1, (), Durability::Sequenced)
    .await
    .unwrap();
assert_eq!(out.base_offset, Some(0));

let batches = engine.fetch(&p, 0, 1024).await.unwrap();
assert_eq!(batches[0].payload, "hello");

Structs§

AppendOutcome
Outcome of a LogEngine::produce call.
AuthoritativePartitionState
Point-in-time durable bounds; grants no exclusivity or pinned history view.
BatchLocation
Where a batch’s bytes live inside its object. Authored by the engine (it owns object layout); the sequencer stores it in the index and returns it from Sequencer::lookup.
BudgetConfig
Configuration for the media-ops budget controller.
BufferStats
Snapshot of the engine’s buffering envelope.
CommitBatch
One batch presented to Sequencer::commit. The engine fills partition, record_count, and location (all engine-visible); meta is forwarded uninterpreted — only the sequencer reads it.
EffectiveKnob
One inspectable knob with layered sources.
FetchPlan
Unforgeable metadata page from one sequencer/blob authority. Reserve the declared metadata budget before constructing, then per-batch bytes before read.
FetchedBatch
A batch read back by LogEngine::fetch, with its assigned base offset.
FlushConfig
Flush-trigger policy for the engine’s group-commit buffer.
InMemorySequencer
A single-node, in-memory Sequencer (Meta = ()): assigns dense offsets and keeps the offset→location index in memory.
IndexEntry
An index entry resolving an offset range to its bytes.
IndexSnapshot
Full index view from ManifestSequencer::snapshot.
LocalBlobStore
Filesystem-backed BlobStore rooted at a directory.
LogEngine
A buffered, multiplexing append-log engine over a BlobStore, with sequencing delegated to a Sequencer.
ManifestSequencer
A Sequencer (Meta = ()) that persists its index to a BlobStore.
MediaOpStats
Durable-media accounting for the flush budget controller (TD-004).
MemoryBlobStore
In-process BlobStore backed by a map. For tests and development; state is lost on drop and is not crash-durable.
MetadataLimits
Explicit source-memory limits for both reads and conditional writes. Transport/header-parser allocations require separate qualification.
MetadataOwners
Separate ownership of in-flight physical work and retained observation bytes. Returned snapshots retain only retained; upload/request sources retain both.
MetadataSnapshot
An opaque observation binding one adapter instance, literal key and version. Payload ownership remains with the snapshot; callers reserve payload, key, token and transport scratch before invoking the physical operation.
PartitionKey
Opaque, engine-visible identifier for an independent offset stream (one dense, monotonic offset space). A Kafka broker maps (topic, partition) onto one of these; a WAL maps a shard. object-log treats it as an opaque key.
PartitionSnapshot
One partition’s index bounds and entries.
PipelineSnapshot
Point-in-time controller + buffer view.

Enums§

BudgetMode
How the engine resolves budget pressure vs admitted latency.
CasOutcome
Result of BlobStore::compare_and_swap.
CommitOutcome
Per-batch result of Sequencer::commit.
ConditionalCommitOutcome
Opt-in conditional results; existing CommitOutcome encoding stays unchanged.
Durability
The durability point a LogEngine::produce call resolves at.
EffectiveReason
Why an effective value was chosen (for inspectability).
MetadataSwap
A definite metadata conditional-write outcome. Transport errors can follow durable publication and must never be interpreted as definite rejection.
ObjectLogError
Errors returned by the object-log engine, storage adapters, and sequencer.

Traits§

BlobStore
A minimal async object store over immutable, string-keyed blobs.
BoundedMetadataStore
Required bounded authority operations, independent of legacy BlobStore. There are deliberately no default implementations. limits bounds every retained payload and token, including conflict observations; transport/header and operation ownership require separate caller admission. Cancellation does not prove a conditional write had no effects. Owners must drive and drain it. Input/output source buffers retain the supplied lifetime owner. This does not attest to all SDK transport allocations or provider-side effects after an ambiguous request; those require separate transport/durability qualification.
BoundedSequencer
Compile-time native page port; legacy custom sequencers need not implement it.
Sequencer
The linearization point: assigns offsets to durably-stored batches and owns the offset→location index.

Functions§

reap_orphans
Delete objects under data_prefix whose keys are absent from live.

Type Aliases§

AppendOwner
An embedding operation lifetime, retained through queued work and started upload completion, including copied upload buffers and failure draining. It must not depend on the caller waiting for its acknowledgement.