Skip to main content

LogEngine

Struct LogEngine 

Source
pub struct LogEngine<S: Sequencer> { /* private fields */ }
Expand description

A buffered, multiplexing append-log engine over a BlobStore, with sequencing delegated to a Sequencer.

produce group-commits: many batches across many partitions are multiplexed into one object, PUT durably, then handed to the sequencer in a single call — so PUT count is decoupled from produce count. A single flush worker preserves per-PartitionKey arrival order and never splits a partition across concurrent commits.

Implementations§

Source§

impl<S> LogEngine<S>
where S: Sequencer + 'static, S::Meta: Send + 'static,

Source

pub fn new( blob: Arc<dyn BlobStore>, sequencer: Arc<S>, config: FlushConfig, key_prefix: impl Into<String>, ) -> Self

Create an engine over blob and sequencer with the given flush policy. Objects are keyed <key_prefix><counter>; pick a prefix unique to this engine instance if several share a store.

Source

pub fn data_prefix(&self) -> &str

Prefix used for data objects written by this engine.

Source

pub fn blob_store(&self) -> &Arc<dyn BlobStore>

Borrow the engine’s blob store (e.g. for orphan reaping or inspection).

Source

pub fn sequencer(&self) -> &Arc<S>

Borrow the engine’s sequencer.

Source

pub async fn produce( &self, partition: PartitionKey, payload: Bytes, record_count: i32, meta: S::Meta, durability: Durability, ) -> Result<AppendOutcome, ObjectLogError>

Buffer a batch and resolve at the requested Durability.

Source

pub async fn flush(&self) -> Result<(), ObjectLogError>

Seal every batch enqueued at or before this call (barrier).

Use after Durability::Buffered produces to wait for durable PUT (and sequencing) of that prior work. Concurrent produces enqueued after this call are not required to finish. Empty buffer returns immediately.

Source

pub async fn fetch( &self, partition: &PartitionKey, offset: i64, max_bytes: usize, ) -> Result<Vec<FetchedBatch>, ObjectLogError>

Read batches covering offsets at/after offset, up to ~max_bytes.

Source

pub async fn fetch_stream<F>( &self, partition: &PartitionKey, offset: i64, visit: F, ) -> Result<(), ObjectLogError>

Stream batches from offset onward without materializing a full Vec.

Calls visit once per index entry in order. Suitable for wide offset windows (bounded-RAM replay). Stops and returns the error if visit fails. Unlike [fetch], there is no byte budget — visit every remaining batch (or stop yourself inside visit).

Source

pub async fn truncate_before( &self, partition: &PartitionKey, offset: i64, ) -> Result<(), ObjectLogError>

Drop the partition’s log below offset and delete any object that thereby becomes fully unreferenced.

Source

pub async fn reap_orphans( &self, live: &HashSet<String>, ) -> Result<Vec<String>, ObjectLogError>

Delete data-prefix objects that are not in live.

Safety: call only when this engine (and any other writer using the same prefix) is quiescent — otherwise an in-flight put that has not yet been committed may be deleted as an “orphan”. Prefer running offline or after drop. Does not touch keys outside data_prefix (e.g. a separate manifest prefix is safe).

Returns the deleted object keys.

Source

pub fn buffer_stats(&self) -> BufferStats

Return a point-in-time snapshot of queued and in-flight payload bytes.

Source

pub fn pipeline_snapshot(&self) -> PipelineSnapshot

Inspect budget controller layers and counters (TD-004).

Trait Implementations§

Source§

impl<S: Sequencer> Drop for LogEngine<S>

Source§

fn drop(&mut self)

Executes the destructor for this type. Read more
Source§

fn pin_drop(self: Pin<&mut Self>)

🔬This is a nightly-only experimental API. (pin_ergonomics)
Execute the destructor for this type, but different to Drop::drop, it requires self to be pinned. Read more

Auto Trait Implementations§

§

impl<S> !RefUnwindSafe for LogEngine<S>

§

impl<S> !UnwindSafe for LogEngine<S>

§

impl<S> Freeze for LogEngine<S>

§

impl<S> Send for LogEngine<S>

§

impl<S> Sync for LogEngine<S>

§

impl<S> Unpin for LogEngine<S>

§

impl<S> UnsafeUnpin for LogEngine<S>

Blanket Implementations§

Source§

impl<T> Any for T
where T: 'static + ?Sized,

Source§

fn type_id(&self) -> TypeId

Gets the TypeId of self. Read more
Source§

impl<T> Borrow<T> for T
where T: ?Sized,

Source§

fn borrow(&self) -> &T

Immutably borrows from an owned value. Read more
Source§

impl<T> BorrowMut<T> for T
where T: ?Sized,

Source§

fn borrow_mut(&mut self) -> &mut T

Mutably borrows from an owned value. Read more
Source§

impl<T> From<T> for T

Source§

fn from(t: T) -> T

Returns the argument unchanged.

Source§

impl<T, U> Into<U> for T
where U: From<T>,

Source§

fn into(self) -> U

Calls U::from(self).

That is, this conversion is whatever the implementation of From<T> for U chooses to do.

Source§

impl<T, U> TryFrom<U> for T
where U: Into<T>,

Source§

type Error = Infallible

The type returned in the event of a conversion error.
Source§

fn try_from(value: U) -> Result<T, <T as TryFrom<U>>::Error>

Performs the conversion.
Source§

impl<T, U> TryInto<U> for T
where U: TryFrom<T>,

Source§

type Error = <U as TryFrom<T>>::Error

The type returned in the event of a conversion error.
Source§

fn try_into(self) -> Result<U, <U as TryFrom<T>>::Error>

Performs the conversion.