Skip to main content

LogEngine

Struct LogEngine 

Source
pub struct LogEngine<S: Sequencer> { /* private fields */ }
Expand description

A buffered, multiplexing append-log engine over a BlobStore, with sequencing delegated to a Sequencer.

produce group-commits: many batches across many partitions are multiplexed into one object, PUT durably, then handed to the sequencer in a single call — so PUT count is decoupled from produce count. A single flush worker preserves per-PartitionKey arrival order and never splits a partition across concurrent commits.

Implementations§

Source§

impl<S> LogEngine<S>
where S: Sequencer + 'static, S::Meta: Send + 'static,

Source

pub fn new( blob: Arc<dyn BlobStore>, sequencer: Arc<S>, config: FlushConfig, key_prefix: impl Into<String>, ) -> Self

Create an engine over blob and sequencer with the given flush policy. Objects are keyed <key_prefix><counter>; pick a prefix unique to this engine instance if several share a store.

Source

pub fn data_prefix(&self) -> &str

Prefix used for data objects written by this engine.

Source

pub fn blob_store(&self) -> &Arc<dyn BlobStore> ⓘ

Borrow the engine’s blob store (e.g. for orphan reaping or inspection).

Source

pub fn sequencer(&self) -> &Arc<S> ⓘ

Borrow the engine’s sequencer.

Source

pub async fn produce( &self, partition: PartitionKey, payload: Bytes, record_count: i32, meta: S::Meta, durability: Durability, ) -> Result<AppendOutcome, ObjectLogError>

Buffer a batch and resolve at the requested Durability.

Source

pub async fn flush(&self) -> Result<(), ObjectLogError>

Seal every batch enqueued at or before this call (barrier).

Use after Durability::Buffered produces to wait for durable PUT (and sequencing) of that prior work. Concurrent produces enqueued after this call are not required to finish. An empty successful buffer returns immediately. A failed PUT or commit remains an error for every barrier covering that enqueue prefix, even after later appends succeed. Reopen the engine to establish a new prefix from its successfully committed index.

Source

pub async fn fetch( &self, partition: &PartitionKey, offset: i64, max_bytes: usize, ) -> Result<Vec<FetchedBatch>, ObjectLogError>

Read batches covering offsets at/after offset, up to ~max_bytes.

Source

pub async fn fetch_stream<F>( &self, partition: &PartitionKey, offset: i64, visit: F, ) -> Result<(), ObjectLogError>

Stream batches from offset onward without materializing a full Vec.

Calls visit once per index entry in order. Suitable for wide offset windows (bounded-RAM replay). Stops and returns the error if visit fails. Unlike Self::fetch, there is no byte budget — visit every remaining batch (or stop yourself inside visit).

Source

pub async fn truncate_before( &self, partition: &PartitionKey, offset: i64, ) -> Result<(), ObjectLogError>

Drop the partition’s log below offset and delete any object that thereby becomes fully unreferenced.

Source

pub async fn reap_orphans( &self, live: &HashSet<String>, ) -> Result<Vec<String>, ObjectLogError>

Delete data-prefix objects that are not in live.

Safety: call only when this engine (and any other writer using the same prefix) is quiescent — otherwise an in-flight put that has not yet been committed may be deleted as an “orphan”. Prefer running offline or after drop. Does not touch keys outside data_prefix (e.g. a separate manifest prefix is safe).

Returns the deleted object keys.

Source

pub fn buffer_stats(&self) -> BufferStats

Return a point-in-time snapshot of queued and in-flight payload bytes.

Source

pub fn pipeline_snapshot(&self) -> PipelineSnapshot

Inspect budget controller layers and counters (TD-004).

Trait Implementations§

Source§

impl<S: Sequencer> Drop for LogEngine<S>

Source§

fn drop(&mut self)

Executes the destructor for this type. Read more
Source§

fn pin_drop(self: Pin<&mut Self>)

🔬This is a nightly-only experimental API. (pin_ergonomics)
Execute the destructor for this type, but different to Drop::drop, it requires self to be pinned. Read more

Auto Trait Implementations§

§

impl<S> !RefUnwindSafe for LogEngine<S>

§

impl<S> !UnwindSafe for LogEngine<S>

§

impl<S> Freeze for LogEngine<S>
where Arc<Shared<<S as Sequencer>::Meta>>: Freeze, Arc<S>: Freeze,

§

impl<S> Send for LogEngine<S>
where Arc<Shared<<S as Sequencer>::Meta>>: Send, Arc<S>: Send,

§

impl<S> Sync for LogEngine<S>
where Arc<Shared<<S as Sequencer>::Meta>>: Sync, Arc<S>: Sync,

§

impl<S> Unpin for LogEngine<S>
where Arc<Shared<<S as Sequencer>::Meta>>: Unpin, Arc<S>: Unpin,

§

impl<S> UnsafeUnpin for LogEngine<S>
where Arc<Shared<<S as Sequencer>::Meta>>: UnsafeUnpin, Arc<S>: UnsafeUnpin,

Blanket Implementations§

Source§

impl<T> Any for T
where T: 'static + ?Sized,

Source§

fn type_id(&self) -> TypeId

Gets the TypeId of self. Read more
Source§

impl<T> Borrow<T> for T
where T: ?Sized,

Source§

fn borrow(&self) -> &T

Immutably borrows from an owned value. Read more
Source§

impl<T> BorrowMut<T> for T
where T: ?Sized,

Source§

fn borrow_mut(&mut self) -> &mut T

Mutably borrows from an owned value. Read more
Source§

impl<T> From<T> for T

Source§

fn from(t: T) -> T

Returns the argument unchanged.

Source§

impl<T, U> Into<U> for T
where U: From<T>,

Source§

fn into(self) -> U

Calls U::from(self).

That is, this conversion is whatever the implementation of From<T> for U chooses to do.

Source§

impl<T, U> TryFrom<U> for T
where U: Into<T>,

Source§

type Error = !

The type returned in the event of a conversion error.
Source§

fn try_from(value: U) -> Result<T, !>

Performs the conversion.
Source§

impl<T, U> TryInto<U> for T
where U: TryFrom<T>,

Source§

type Error = <U as TryFrom<T>>::Error

The type returned in the event of a conversion error.
Source§

fn try_into(self) -> Result<U, <U as TryFrom<T>>::Error>

Performs the conversion.