Struct SpilledHashAgg 

pub struct SpilledHashAgg<K, S>
where
    K: Hash + Eq + Clone + SpillCodec,
    S: Clone + Mergeable + SpillCodec,
{
    pub total_spilled_bytes: u64,
    pub spill_count: u64,
    /* private fields */
}

Hash aggregation table with optional spill-to-disk overflow.

Owns the in-memory HashMap<K, S> plus a list of PathBufs pointing at spilled batch files. The caller drives the lifecycle by calling accumulate for each input row and drain once the input is exhausted.

Fields

total_spilled_bytes: u64

Total bytes spilled across all batches — diagnostic.

spill_count: u64

Number of times spill_partition was called — diagnostic.

Implementations

impl<K, S> SpilledHashAgg<K, S>

where K: Hash + Eq + Clone + SpillCodec, S: Clone + Mergeable + SpillCodec,

pub fn new( spill_dir: impl AsRef<Path>, mem_limit_bytes: usize, avg_entry_bytes: usize, ) -> SpilledHashAgg<K, S>

Create a new spillable hash aggregator. spill_dir must exist and be writable; the helper does NOT create it. mem_limit_bytes == 0 disables spilling entirely (useful for tests that want to exercise the in-memory path).

pub fn accumulate(&mut self, key: K, increment: S) -> Result<(), SpillError>

Insert or update an aggregation state for the given key. accumulate triggers a spill when the in-memory table’s estimated footprint exceeds the configured limit. Returns the key/state pair after the merge so callers can chain.

pub fn spill_partition(&mut self) -> Result<(), SpillError>

Write the entire in-memory table to a new spill batch file and clear the table. Updates the spill diagnostics. Caller is free to keep accumulating after this returns — the batch will be merged back during drain.

pub fn drain(self) -> Result<HashMap<K, S>, SpillError>

Consume the aggregator and return the final merged state for every group. Reads every spilled batch back into a new in-memory hash table, merges with whatever the accumulator left in place, and yields the unified set.

Memory profile: at peak, this holds ONE spill batch plus the running merge table in memory simultaneously. If a single spill batch is larger than mem_limit_bytes, we return BatchTooLarge so the caller can switch strategies.

pub fn spilled_batch_count(&self) -> usize

Number of spill batches currently on disk. Diagnostic hook for tests / metrics.

pub fn in_memory_groups(&self) -> usize

Number of groups currently held in memory.

Trait Implementations

impl<K, S> Drop for SpilledHashAgg<K, S>

where K: Hash + Eq + Clone + SpillCodec, S: Clone + Mergeable + SpillCodec,

fn drop(&mut self)

Executes the destructor for this type.

fn pin_drop(self: Pin<&mut Self>)

🔬This is a nightly-only experimental API. (pin_ergonomics)

Execute the destructor for this type, but different to Drop::drop, it requires self to be pinned.