object-log
A buffered, multiplexing append log over pluggable object storage.
object-log stores an ordered, offset-addressed log as immutable objects in any
BlobStore (memory, local filesystem, or — behind the s3 feature — S3). It
deals only in opaque payload bytes: it knows nothing about record formats,
Kafka, or brokers. Many produce calls group-commit into one object, so PUT count
is decoupled from produce count. It is the storage engine extracted from the
fjord Kafka-compatible broker.
Highlights
BlobStoreport — minimal async trait with durable-on-return writes.LocalBlobStore: temp →sync_data→ rename → dirfsync; streamingput_chunks. OptionalS3BlobStore(s3feature).LogEngine— group-commit: manyproducecalls → one object. Ack viaDurability::{Buffered,Durable,Sequenced}or pipeline +flush(). Linger packs under load (default 50 ms);max_bytesdefault 1 GiB is only a safety ceiling. Default-on durable-ops budget; early-flush only when idle (pipeline_snapshot()).Sequencerseam — offsets + index;InMemorySequencerandManifestSequencer. Engine forwardsMetauninterpreted.
Performance check
OBJECT_LOG_PERF_BYTES= \
Prints dd / B0 / B1 / B2 (zeros, fair timers). See TD-004.
Usage
[]
= "0.2"
use ;
use Bytes;
use Arc;
let engine = new;
let p = PartitionKey;
let out = engine
.produce
.await?;
assert_eq!;
let read = engine.fetch.await?;
assert_eq!;
(A runnable version is the crate-level doctest — see docs.rs.)
To target S3/Garage/MinIO, enable the s3 feature and use S3BlobStore, or
implement the BlobStore trait for your client.
Status
0.2.x — pre-1.0; the API may evolve. Requires Rust 1.88+ (edition 2024).
License
Licensed under either of Apache License, Version 2.0 or MIT license at your option. Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in this crate by you, as defined in the Apache-2.0 license, shall be dual licensed as above, without any additional terms or conditions.