Struct Collector 

pub struct Collector { /* private fields */ }

Reads batches of ingested entries from object storage via a queue consumer.

The collector iterates over entries in the queue manifest in ingestion order, fetches the corresponding data batches from object storage, and makes them available to the caller. Epoch-based fencing ensures only a single active collector processes entries at any time.

Implementations

impl Collector

pub fn new(config: CollectorConfig) -> Result<Self>

Create a new collector from the given configuration.

pub async fn initialize(&self, last_acked_sequence: Option<u64>) -> Result<()>

Initialize the consumer by fencing any previous consumer instance.

If last_acked_sequence is Some(seq), the collector resumes after that sequence — the next call to next_batch will read sequence seq + 1. If None, the collector peeks the queue to discover the first available entry and positions itself just before it.

pub async fn next_batch(&self) -> Result<Option<CollectedBatch>>

Read the next data batch from object storage.

If no batch has been acked yet, peeks the earliest entry in the queue. Otherwise, reads the entry with sequence last_acked_sequence + 1. Returns None if no matching entry is available.

pub async fn ack(&self, sequence: u64) -> Result<()>

Acknowledge that the batch with the given sequence number has been processed.

Acks must be in order — the sequence must immediately follow the last acked sequence, otherwise an error is returned. To amortize manifest writes, the collector only calls dequeue() on the queue consumer every 100 acks.

pub async fn flush(&self) -> Result<()>

Flush any pending acks by dequeueing up to the last acked sequence.

pub async fn close(self) -> Result<()>

Flush pending acks and consume the collector.

pub fn len(&self) -> usize

Return the number of entries in the queue as of the last manifest read or write.

pub fn is_empty(&self) -> bool

Return true if the queue had no entries as of the last manifest read or write.

pub fn conflict_rate(&self) -> f64

Return the percentage of manifest writes that encountered a conflict.