Module unspillable_headroom 

A memory pool that keeps a slice of the budget out of reach of spillable consumers, so an operator that cannot spill can still make progress.

The failure this exists for

FairSpillPool treats its two consumer classes asymmetrically. A spillable consumer is capped at its fair share of pool_size - unspillable; an unspillable one gets whatever is left after both classes:

// datafusion-execution-54/src/memory_pool/pool.rs, FairSpillPool::try_grow
false => {
    let available = self.pool_size
        .saturating_sub(state.unspillable + state.spillable);

So N spillable consumers, each politely inside its own pool/N share, can between them occupy the entire pool — and the pool has no way to make any of them give it back. Spilling in DataFusion is driven by a consumer’s own try_grow failing; there is no callback the pool can invoke to reclaim. The next unspillable consumer to ask for memory therefore gets zero, however little it wants.

TPC-H q10 and q11 at SF100 died exactly there:

Failed to allocate additional 877.0 B for HashJoinInput with 0.0 B already
allocated for this reservation - 0.0 B remain available for the total
memory pool: fair(pool_size: 2.3 GB)

877 bytes, refused by a 2.3 GB pool. A hash join build side cannot spill (DataFusion has no spilling HashJoinExec), so these are the small joins crate::spillable_join::SpillableJoinSelection correctly declined to convert — they need very little, and there was nothing left to give.

What this does

Bounds the total spillable footprint at pool_size - headroom, leaving headroom that only unspillable consumers can occupy. Spillable consumers hit their ceiling earlier and spill, which is the behaviour they are built for and already exercise; unspillable ones keep a floor they can always draw on. The fair-share rule between spillable consumers is unchanged — that is still FairSpillPool’s job, and this delegates to it.

This is a ceiling on spillers, not a reservation: when no unspillable consumer is running, the headroom simply goes unused, which costs a query that spills slightly earlier than it strictly had to. That is the trade — a little more spilling against queries that cannot run at all.

Structs

UnspillableHeadroomPool

See the module docs.

Constants

DEFAULT_UNSPILLABLE_HEADROOM_DENOMINATOR

Denominator of DEFAULT_UNSPILLABLE_HEADROOM_NUMERATOR.

DEFAULT_UNSPILLABLE_HEADROOM_NUMERATOR

Fraction of the pool held back for consumers that cannot spill.

UNSPILLABLE_HEADROOM_PERCENT_ENV

Environment override for the headroom, as a percentage of the pool.

Functions

headroom_bytes

Headroom in bytes for a pool of pool_size, honouring the env override.