pub struct LogEngine<S: Sequencer> { /* private fields */ }Expand description
A buffered, multiplexing append-log engine over a BlobStore, with
sequencing delegated to a Sequencer.
produce group-commits: many batches across many partitions are multiplexed
into one object, PUT durably, then handed to the sequencer in a single call —
so PUT count is decoupled from produce count. A single flush worker preserves
per-PartitionKey arrival order and never splits a partition across
concurrent commits.
Implementations§
Source§impl<S> LogEngine<S>
impl<S> LogEngine<S>
Sourcepub fn new(
blob: Arc<dyn BlobStore>,
sequencer: Arc<S>,
config: FlushConfig,
key_prefix: impl Into<String>,
) -> Self
pub fn new( blob: Arc<dyn BlobStore>, sequencer: Arc<S>, config: FlushConfig, key_prefix: impl Into<String>, ) -> Self
Create an engine over blob and sequencer with the given flush policy.
Objects are keyed <key_prefix><counter>; pick a prefix unique to this
engine instance if several share a store.
Sourcepub fn data_prefix(&self) -> &str
pub fn data_prefix(&self) -> &str
Prefix used for data objects written by this engine.
Sourcepub fn blob_store(&self) -> &Arc<dyn BlobStore>
pub fn blob_store(&self) -> &Arc<dyn BlobStore>
Borrow the engine’s blob store (e.g. for orphan reaping or inspection).
Sourcepub async fn produce(
&self,
partition: PartitionKey,
payload: Bytes,
record_count: i32,
meta: S::Meta,
durability: Durability,
) -> Result<AppendOutcome, ObjectLogError>
pub async fn produce( &self, partition: PartitionKey, payload: Bytes, record_count: i32, meta: S::Meta, durability: Durability, ) -> Result<AppendOutcome, ObjectLogError>
Buffer a batch and resolve at the requested Durability.
Sourcepub async fn flush(&self) -> Result<(), ObjectLogError>
pub async fn flush(&self) -> Result<(), ObjectLogError>
Seal every batch enqueued at or before this call (barrier).
Use after Durability::Buffered produces to wait for durable PUT (and
sequencing) of that prior work. Concurrent produces enqueued after this
call are not required to finish. Empty buffer returns immediately.
Sourcepub async fn fetch(
&self,
partition: &PartitionKey,
offset: i64,
max_bytes: usize,
) -> Result<Vec<FetchedBatch>, ObjectLogError>
pub async fn fetch( &self, partition: &PartitionKey, offset: i64, max_bytes: usize, ) -> Result<Vec<FetchedBatch>, ObjectLogError>
Read batches covering offsets at/after offset, up to ~max_bytes.
Sourcepub async fn fetch_stream<F>(
&self,
partition: &PartitionKey,
offset: i64,
visit: F,
) -> Result<(), ObjectLogError>
pub async fn fetch_stream<F>( &self, partition: &PartitionKey, offset: i64, visit: F, ) -> Result<(), ObjectLogError>
Stream batches from offset onward without materializing a full Vec.
Calls visit once per index entry in order. Suitable for wide offset
windows (bounded-RAM replay). Stops and returns the error if visit
fails. Unlike [fetch], there is no byte budget — visit every remaining
batch (or stop yourself inside visit).
Sourcepub async fn truncate_before(
&self,
partition: &PartitionKey,
offset: i64,
) -> Result<(), ObjectLogError>
pub async fn truncate_before( &self, partition: &PartitionKey, offset: i64, ) -> Result<(), ObjectLogError>
Drop the partition’s log below offset and delete any object that thereby
becomes fully unreferenced.
Sourcepub async fn reap_orphans(
&self,
live: &HashSet<String>,
) -> Result<Vec<String>, ObjectLogError>
pub async fn reap_orphans( &self, live: &HashSet<String>, ) -> Result<Vec<String>, ObjectLogError>
Delete data-prefix objects that are not in live.
Safety: call only when this engine (and any other writer using the
same prefix) is quiescent — otherwise an in-flight put that has not yet
been committed may be deleted as an “orphan”. Prefer running offline or
after drop. Does not touch keys outside data_prefix
(e.g. a separate manifest prefix is safe).
Returns the deleted object keys.
Sourcepub fn buffer_stats(&self) -> BufferStats
pub fn buffer_stats(&self) -> BufferStats
Return a point-in-time snapshot of queued and in-flight payload bytes.
Sourcepub fn pipeline_snapshot(&self) -> PipelineSnapshot
pub fn pipeline_snapshot(&self) -> PipelineSnapshot
Inspect budget controller layers and counters (TD-004).