pub struct LogEngine<S: Sequencer> { /* private fields */ }Expand description
A buffered, multiplexing append-log engine over a BlobStore, with
sequencing delegated to a Sequencer.
produce group-commits: many batches across many partitions are multiplexed
into one object, PUT durably, then handed to the sequencer in a single call —
so PUT count is decoupled from produce count. A single flush worker preserves
per-PartitionKey arrival order and never splits a partition across
concurrent commits.
Implementations§
Source§impl<S> LogEngine<S>
impl<S> LogEngine<S>
Sourcepub fn new(
blob: Arc<dyn BlobStore>,
sequencer: Arc<S>,
config: FlushConfig,
key_prefix: impl Into<String>,
) -> Self
pub fn new( blob: Arc<dyn BlobStore>, sequencer: Arc<S>, config: FlushConfig, key_prefix: impl Into<String>, ) -> Self
Create an engine over blob and sequencer with the given flush policy.
Objects are keyed <key_prefix><counter>; pick a prefix unique to this
engine instance if several share a store.
Sourcepub fn new_with_writer(
blob: Arc<dyn BlobStore>,
sequencer: Arc<S>,
config: FlushConfig,
key_prefix: impl Into<String>,
writer_id: impl Into<String>,
) -> Self
pub fn new_with_writer( blob: Arc<dyn BlobStore>, sequencer: Arc<S>, config: FlushConfig, key_prefix: impl Into<String>, writer_id: impl Into<String>, ) -> Self
Like new, but data objects are
<key_prefix><writer_id>/<epoch>/<counter> so two writers cannot
overwrite the same key. writer_id must not contain / or ...
Sourcepub fn data_prefix(&self) -> &str
pub fn data_prefix(&self) -> &str
Prefix used for data objects written by this engine.
Sourcepub fn blob_store(&self) -> &Arc<dyn BlobStore> ⓘ
pub fn blob_store(&self) -> &Arc<dyn BlobStore> ⓘ
Borrow the engine’s blob store (e.g. for orphan reaping or inspection).
Sourcepub fn writer_id(&self) -> &str
pub fn writer_id(&self) -> &str
Writer id used in data-object keys. Empty when constructed with new.
Sourcepub async fn produce(
&self,
partition: PartitionKey,
payload: Bytes,
record_count: i32,
meta: S::Meta,
durability: Durability,
) -> Result<AppendOutcome, ObjectLogError>
pub async fn produce( &self, partition: PartitionKey, payload: Bytes, record_count: i32, meta: S::Meta, durability: Durability, ) -> Result<AppendOutcome, ObjectLogError>
Buffer a batch and resolve at the requested Durability.
The batch’s fence epoch is 0. Use produce_at_epoch
when the partition index is fenced.
Sourcepub async fn produce_at_epoch(
&self,
partition: PartitionKey,
payload: Bytes,
record_count: i32,
meta: S::Meta,
durability: Durability,
epoch: u64,
) -> Result<AppendOutcome, ObjectLogError>
pub async fn produce_at_epoch( &self, partition: PartitionKey, payload: Bytes, record_count: i32, meta: S::Meta, durability: Durability, epoch: u64, ) -> Result<AppendOutcome, ObjectLogError>
produce at a caller-supplied fence epoch.
Sourcepub async fn produce_owned_at_epoch(
&self,
partition: PartitionKey,
payload: Bytes,
record_count: i32,
meta: S::Meta,
durability: Durability,
epoch: u64,
owner: AppendOwner,
) -> Result<AppendOutcome, ObjectLogError>
pub async fn produce_owned_at_epoch( &self, partition: PartitionKey, payload: Bytes, record_count: i32, meta: S::Meta, durability: Durability, epoch: u64, owner: AppendOwner, ) -> Result<AppendOutcome, ObjectLogError>
Append with an operation lifetime independent of caller cancellation. Acknowledgement is not a terminal-owner notification: use the owner’s final Drop/callback to observe release of all physical work.
Sourcepub async fn produce_conditional_at_epoch(
&self,
partition: PartitionKey,
payload: Bytes,
record_count: i32,
meta: S::Meta,
epoch: u64,
expected_head: i64,
owner: AppendOwner,
) -> Result<AppendOutcome, ObjectLogError>
pub async fn produce_conditional_at_epoch( &self, partition: PartitionKey, payload: Bytes, record_count: i32, meta: S::Meta, epoch: u64, expected_head: i64, owner: AppendOwner, ) -> Result<AppendOutcome, ObjectLogError>
Owned compare-and-append; acknowledgement requires a sequencing decision.
Sourcepub async fn flush(&self) -> Result<(), ObjectLogError>
pub async fn flush(&self) -> Result<(), ObjectLogError>
Seal every batch enqueued at or before this call (barrier).
Use after Durability::Buffered produces to wait for durable PUT (and
sequencing) of that prior work. Concurrent produces enqueued after this
call are not required to finish. An empty successful buffer returns immediately.
A failed PUT or commit remains an error for every barrier covering that
enqueue prefix, even after later appends succeed. Reopen the engine to
establish a new prefix from its successfully committed index.
Sourcepub async fn fetch(
&self,
partition: &PartitionKey,
offset: i64,
max_bytes: usize,
) -> Result<Vec<FetchedBatch>, ObjectLogError>
pub async fn fetch( &self, partition: &PartitionKey, offset: i64, max_bytes: usize, ) -> Result<Vec<FetchedBatch>, ObjectLogError>
Read batches covering offsets at/after offset, up to ~max_bytes.
Sourcepub async fn fetch_stream<F>(
&self,
partition: &PartitionKey,
offset: i64,
visit: F,
) -> Result<(), ObjectLogError>
pub async fn fetch_stream<F>( &self, partition: &PartitionKey, offset: i64, visit: F, ) -> Result<(), ObjectLogError>
Stream batches from offset onward without materializing a full Vec.
Calls visit once per index entry in order. Suitable for wide offset
windows (bounded-RAM replay). Stops and returns the error if visit
fails. Unlike Self::fetch, there is no byte budget — visit every remaining
batch (or stop yourself inside visit).
Sourcepub async fn truncate_before(
&self,
partition: &PartitionKey,
offset: i64,
) -> Result<(), ObjectLogError>
pub async fn truncate_before( &self, partition: &PartitionKey, offset: i64, ) -> Result<(), ObjectLogError>
Drop the partition’s log below offset and delete any object that thereby
becomes fully unreferenced.
Sourcepub async fn reap_orphans(
&self,
live: &HashSet<String>,
) -> Result<Vec<String>, ObjectLogError>
pub async fn reap_orphans( &self, live: &HashSet<String>, ) -> Result<Vec<String>, ObjectLogError>
Delete data-prefix objects that are not in live.
Safety: call only when this engine (and any other writer using the
same prefix) is quiescent — otherwise an in-flight put that has not yet
been committed may be deleted as an “orphan”. Prefer running offline or
after drop. Does not touch keys outside data_prefix
(e.g. a separate manifest prefix is safe).
Returns the deleted object keys.
Sourcepub fn buffer_stats(&self) -> BufferStats
pub fn buffer_stats(&self) -> BufferStats
Return a point-in-time snapshot of queued and in-flight payload bytes.
Sourcepub fn pipeline_snapshot(&self) -> PipelineSnapshot
pub fn pipeline_snapshot(&self) -> PipelineSnapshot
Inspect budget controller layers and counters (TD-004).
Source§impl<S: BoundedSequencer + 'static> LogEngine<S>
impl<S: BoundedSequencer + 'static> LogEngine<S>
Sourcepub fn plan_fetch(
&self,
partition: &PartitionKey,
offset: i64,
max_entries: usize,
max_metadata_bytes: usize,
target_payload_bytes: u64,
) -> Result<FetchPlan<S>, ObjectLogError>
pub fn plan_fetch( &self, partition: &PartitionKey, offset: i64, max_entries: usize, max_metadata_bytes: usize, target_payload_bytes: u64, ) -> Result<FetchPlan<S>, ObjectLogError>
Plan a count/metadata-bounded page without reading payloads. The soft target includes at least one batch; use a separate hard admitted bound.
Sourcepub async fn read_planned_batch(
&self,
plan: &FetchPlan<S>,
index: usize,
max_bytes: u64,
) -> Result<FetchedBatch, ObjectLogError>
pub async fn read_planned_batch( &self, plan: &FetchPlan<S>, index: usize, max_bytes: u64, ) -> Result<FetchedBatch, ObjectLogError>
Caller admits entry.location.byte_len before this read. max_bytes is
a hard batch bound; a soft target may admit an oversized first batch.