Struct Consumer 

pub struct Consumer { /* private fields */ }

Reads batches of ingested entries from object storage via a queue consumer.

The consumer iterates over entries in the queue manifest in ingestion order, fetches the corresponding data batches from object storage, and makes them available to the caller. Epoch-based fencing ensures only a single active consumer processes entries at any time.

Implementations

impl Consumer

pub async fn new( config: ConsumerConfig, last_acked_sequence: Option<u64>, ) -> Result<Self>

Create a new consumer from the given configuration.

Initializes the queue consumer (fencing any previous instance) and spawns the garbage collector. If last_acked_sequence is Some(seq), the consumer resumes after that sequence; if None, it discovers the first available entry.

pub async fn with_object_store( config: ConsumerConfig, object_store: Arc<dyn ObjectStore>, last_acked_sequence: Option<u64>, ) -> Result<Self>

pub async fn next_batch(&mut self) -> Result<Option<ConsumedBatch>>

Read the next data batch from object storage.

If no batch has been fetched yet, peeks the earliest entry in the queue. Otherwise, reads the entry with sequence last_fetched_sequence + 1. Returns None if no matching entry is available.

pub async fn ack(&mut self, sequence: u64) -> Result<()>

Acknowledge that the batch with the given sequence number has been processed.

Acks must be in order — the sequence must immediately follow the last acked sequence, otherwise an error is returned. To amortize manifest writes, the consumer only calls dequeue() on the queue consumer every 100 acks.

pub async fn flush(&mut self) -> Result<()>

Flush any pending acks by dequeueing up to the last acked sequence.

pub async fn close(self) -> Result<()>

Flush pending acks, shut down the garbage collector, and consume the handle.

pub fn len(&self) -> usize

Return the number of entries in the queue as of the last manifest read or write.

pub fn is_empty(&self) -> bool

Return true if the queue had no entries as of the last manifest read or write.

pub fn conflict_rate(&self) -> f64

Return the percentage of manifest writes that encountered a conflict.