Struct datafusion::execution::memory_pool::FairSpillPool

pub struct FairSpillPool { /* private fields */ }

A MemoryPool that prevents spillable reservations from using more than an even fraction of the available memory sans any unspillable reservations (i.e. (pool_size - unspillable_memory) / num_spillable_reservations)

This pool works best when you know beforehand the query has multiple spillable operators that will likely all need to spill. Sometimes it will cause spills even when there was sufficient memory (reserved for other operators) to avoid doing so.

   ┌───────────────────────z──────────────────────z───────────────┐
   │                       z                      z               │
   │                       z                      z               │
   │       Spillable       z       Unspillable    z     Free      │
   │        Memory         z        Memory        z    Memory     │
   │                       z                      z               │
   │                       z                      z               │
   └───────────────────────z──────────────────────z───────────────┘

Unspillable memory is allocated in a first-come, first-serve fashion

Implementations

impl FairSpillPool

pub fn new(pool_size: usize) -> FairSpillPool

Allocate up to limit bytes

Trait Implementations

impl Debug for FairSpillPool

fn fmt(&self, f: &mut Formatter<'_>) -> Result<(), Error>

Formats the value using the given formatter.

impl MemoryPool for FairSpillPool

fn register(&self, consumer: &MemoryConsumer)

Registers a new MemoryConsumer

fn unregister(&self, consumer: &MemoryConsumer)

Records the destruction of a MemoryReservation with MemoryConsumer

fn grow(&self, reservation: &MemoryReservation, additional: usize)

Infallibly grow the provided reservation by additional bytes

fn shrink(&self, reservation: &MemoryReservation, shrink: usize)

Infallibly shrink the provided reservation by shrink bytes

fn try_grow( &self, reservation: &MemoryReservation, additional: usize ) -> Result<(), DataFusionError>

Attempt to grow the provided reservation by additional bytes

fn reserved(&self) -> usize

Return the total amount of memory reserved