Struct StreamingFetcher 

pub struct StreamingFetcher { /* private fields */ }

Streaming fetcher for large datasets with checkpoint support

Implementations

impl StreamingFetcher

pub async fn new(config: Config) -> Result<Self>

Create a streaming fetcher

pub fn with_checkpoint(self, path: &Path) -> Result<Self>

Enable checkpointing

pub fn pool(&self) -> &RpcPool

Get the RPC pool

pub fn endpoint_count(&self) -> usize

Get endpoint count

pub async fn fetch_streaming<F>(&mut self, handler: F) -> Result<FetchStats>

where F: FnMut(FetchResult) -> Result<()>,

Fetch logs with streaming output, calling handler for each chunk Returns aggregated stats

Output Order

Important: Chunks are processed in completion order, not block order. This means logs from later blocks may be written before logs from earlier blocks. This is a design choice to maximize throughput - using ordered streams would significantly reduce parallelism benefits.

If you need block-ordered output:

• Use LogFetcher::fetch_all() which sorts results before returning
• Or post-process the output file to sort by block number

Uses parallel fetching with buffer_unordered for improved performance while maintaining sequential handler calls for checkpoint consistency.

pub async fn stream(self, tx: Sender<Result<FetchResult>>) -> Result<FetchStats>

Stream logs through a bounded channel (for async consumers)

Note: Uses try_send to avoid potential deadlock with blocking_send. If channel is full, returns an error. For guaranteed delivery, use stream_unbounded instead.

pub async fn stream_unbounded( self, tx: UnboundedSender<Result<FetchResult>>, ) -> Result<FetchStats>

Stream logs through an unbounded channel (for async consumers)

RUST-003 fix: This variant uses an unbounded channel to avoid potential deadlock issues. The unbounded channel will never block on send, but memory usage is unbounded if the receiver is slow.