pub struct LogEngine<S: Sequencer> { /* private fields */ }Expand description
A buffered, multiplexing append-log engine over a BlobStore, with
sequencing delegated to a Sequencer.
produce group-commits: many batches across many partitions are multiplexed
into one object, PUT durably, then handed to the sequencer in a single call —
so PUT count is decoupled from produce count. A single flush worker preserves
per-PartitionKey arrival order and never splits a partition across
concurrent commits.
Implementations§
Source§impl<S> LogEngine<S>
impl<S> LogEngine<S>
Sourcepub fn new(
blob: Arc<dyn BlobStore>,
sequencer: Arc<S>,
config: FlushConfig,
key_prefix: impl Into<String>,
) -> Self
pub fn new( blob: Arc<dyn BlobStore>, sequencer: Arc<S>, config: FlushConfig, key_prefix: impl Into<String>, ) -> Self
Create an engine over blob and sequencer with the given flush policy.
Objects are keyed <key_prefix><counter>; pick a prefix unique to this
engine instance if several share a store.
Sourcepub fn data_prefix(&self) -> &str
pub fn data_prefix(&self) -> &str
Prefix used for data objects written by this engine.
Sourcepub fn blob_store(&self) -> &Arc<dyn BlobStore> ⓘ
pub fn blob_store(&self) -> &Arc<dyn BlobStore> ⓘ
Borrow the engine’s blob store (e.g. for orphan reaping or inspection).
Sourcepub async fn produce(
&self,
partition: PartitionKey,
payload: Bytes,
record_count: i32,
meta: S::Meta,
durability: Durability,
) -> Result<AppendOutcome, ObjectLogError>
pub async fn produce( &self, partition: PartitionKey, payload: Bytes, record_count: i32, meta: S::Meta, durability: Durability, ) -> Result<AppendOutcome, ObjectLogError>
Buffer a batch and resolve at the requested Durability.
Sourcepub async fn flush(&self) -> Result<(), ObjectLogError>
pub async fn flush(&self) -> Result<(), ObjectLogError>
Seal every batch enqueued at or before this call (barrier).
Use after Durability::Buffered produces to wait for durable PUT (and
sequencing) of that prior work. Concurrent produces enqueued after this
call are not required to finish. An empty successful buffer returns immediately.
A failed PUT or commit remains an error for every barrier covering that
enqueue prefix, even after later appends succeed. Reopen the engine to
establish a new prefix from its successfully committed index.
Sourcepub async fn fetch(
&self,
partition: &PartitionKey,
offset: i64,
max_bytes: usize,
) -> Result<Vec<FetchedBatch>, ObjectLogError>
pub async fn fetch( &self, partition: &PartitionKey, offset: i64, max_bytes: usize, ) -> Result<Vec<FetchedBatch>, ObjectLogError>
Read batches covering offsets at/after offset, up to ~max_bytes.
Sourcepub async fn fetch_stream<F>(
&self,
partition: &PartitionKey,
offset: i64,
visit: F,
) -> Result<(), ObjectLogError>
pub async fn fetch_stream<F>( &self, partition: &PartitionKey, offset: i64, visit: F, ) -> Result<(), ObjectLogError>
Stream batches from offset onward without materializing a full Vec.
Calls visit once per index entry in order. Suitable for wide offset
windows (bounded-RAM replay). Stops and returns the error if visit
fails. Unlike Self::fetch, there is no byte budget — visit every remaining
batch (or stop yourself inside visit).
Sourcepub async fn truncate_before(
&self,
partition: &PartitionKey,
offset: i64,
) -> Result<(), ObjectLogError>
pub async fn truncate_before( &self, partition: &PartitionKey, offset: i64, ) -> Result<(), ObjectLogError>
Drop the partition’s log below offset and delete any object that thereby
becomes fully unreferenced.
Sourcepub async fn reap_orphans(
&self,
live: &HashSet<String>,
) -> Result<Vec<String>, ObjectLogError>
pub async fn reap_orphans( &self, live: &HashSet<String>, ) -> Result<Vec<String>, ObjectLogError>
Delete data-prefix objects that are not in live.
Safety: call only when this engine (and any other writer using the
same prefix) is quiescent — otherwise an in-flight put that has not yet
been committed may be deleted as an “orphan”. Prefer running offline or
after drop. Does not touch keys outside data_prefix
(e.g. a separate manifest prefix is safe).
Returns the deleted object keys.
Sourcepub fn buffer_stats(&self) -> BufferStats
pub fn buffer_stats(&self) -> BufferStats
Return a point-in-time snapshot of queued and in-flight payload bytes.
Sourcepub fn pipeline_snapshot(&self) -> PipelineSnapshot
pub fn pipeline_snapshot(&self) -> PipelineSnapshot
Inspect budget controller layers and counters (TD-004).