pub struct LogEngine<S: Sequencer> { /* private fields */ }Expand description
A buffered, multiplexing append-log engine over a BlobStore, with
sequencing delegated to a Sequencer.
produce group-commits: many batches across many partitions are multiplexed
into one object, PUT durably, then handed to the sequencer in a single call —
so PUT count is decoupled from produce count. A single flush worker preserves
per-PartitionKey arrival order and never splits a partition across
concurrent commits.
Implementations§
Source§impl<S> LogEngine<S>
impl<S> LogEngine<S>
Sourcepub fn new(
blob: Arc<dyn BlobStore>,
sequencer: Arc<S>,
config: FlushConfig,
key_prefix: impl Into<String>,
) -> Self
pub fn new( blob: Arc<dyn BlobStore>, sequencer: Arc<S>, config: FlushConfig, key_prefix: impl Into<String>, ) -> Self
Create an engine over blob and sequencer with the given flush policy.
Objects are keyed <key_prefix><counter>; pick a prefix unique to this
engine instance if several share a store.
Sourcepub async fn produce(
&self,
partition: PartitionKey,
payload: Bytes,
record_count: i32,
meta: S::Meta,
durability: Durability,
) -> Result<AppendOutcome, ObjectLogError>
pub async fn produce( &self, partition: PartitionKey, payload: Bytes, record_count: i32, meta: S::Meta, durability: Durability, ) -> Result<AppendOutcome, ObjectLogError>
Buffer a batch and resolve at the requested Durability.
Sourcepub async fn flush(&self) -> Result<(), ObjectLogError>
pub async fn flush(&self) -> Result<(), ObjectLogError>
Seal every batch enqueued at or before this call (barrier).
Use after Durability::Buffered produces to wait for durable PUT (and
sequencing) of that prior work. Concurrent produces enqueued after this
call are not required to finish. Empty buffer returns immediately.
Sourcepub async fn fetch(
&self,
partition: &PartitionKey,
offset: i64,
max_bytes: usize,
) -> Result<Vec<FetchedBatch>, ObjectLogError>
pub async fn fetch( &self, partition: &PartitionKey, offset: i64, max_bytes: usize, ) -> Result<Vec<FetchedBatch>, ObjectLogError>
Read batches covering offsets at/after offset, up to ~max_bytes.
Sourcepub async fn truncate_before(
&self,
partition: &PartitionKey,
offset: i64,
) -> Result<(), ObjectLogError>
pub async fn truncate_before( &self, partition: &PartitionKey, offset: i64, ) -> Result<(), ObjectLogError>
Drop the partition’s log below offset and delete any object that thereby
becomes fully unreferenced.
Sourcepub fn buffer_stats(&self) -> BufferStats
pub fn buffer_stats(&self) -> BufferStats
Return a point-in-time snapshot of queued and in-flight payload bytes.
Sourcepub fn pipeline_snapshot(&self) -> PipelineSnapshot
pub fn pipeline_snapshot(&self) -> PipelineSnapshot
Inspect budget controller layers and counters (TD-004).