Module streaming_iterator 

Streaming Iterator Architecture for Scans

This module implements memory-efficient streaming iterators that:

• Use O(k) memory for LIMIT k queries regardless of table size
• Enable zero-copy SSTable reads via mmap
• Support merge iteration over multiple sorted sources (LSM-tree style)
• Provide backpressure to prevent memory exhaustion

Memory Complexity

Current scan: M = O(N) where N = rows in scan range Streaming: M = O(k + S × log(S)) where k = LIMIT, S = number of sources

For 100M rows with LIMIT 10:

• Current: 10GB allocation (100M × 100 bytes)
• Streaming: ~10KB (10 rows + heap overhead)

Structs

Entry

A key-value entry with timestamp for MVCC

FilterIterator

Filter iterator for predicate pushdown

LimitIterator

Limit iterator wrapper

MemtableIterator

Memtable iterator (in-memory sorted data)

MergeIterator

Merge iterator over multiple sorted sources (LSM-tree style)

RangeIterator

Range-bounded iterator wrapper

ScanStats

Scan statistics

SstBuilder

Builder for SSTable data (for testing)

SstIterator

SSTable iterator (simulated - real impl would use mmap)

VecIterator

Vector-based entry iterator for testing and simple cases

Traits

EntryIterator

Trait for a streaming source of entries

IteratorExt

Extension trait for iterators