Trait datafusion_execution::memory_pool::MemoryPool

source · 
pub trait MemoryPool: Send + Sync + Debug {
    // Required methods
    fn grow(&self, reservation: &MemoryReservation, additional: usize);
    fn shrink(&self, reservation: &MemoryReservation, shrink: usize);
    fn try_grow(
        &self,
        reservation: &MemoryReservation,
        additional: usize
    ) -> Result<()>;
    fn reserved(&self) -> usize;

    // Provided methods
    fn register(&self, _consumer: &MemoryConsumer) { ... }
    fn unregister(&self, _consumer: &MemoryConsumer) { ... }
}
Expand description

The pool of memory on which MemoryReservations record their memory reservations.

DataFusion is a streaming query engine, processing most queries without buffering the entire input. However, certain operations such as sorting and grouping/joining with a large number of distinct groups/keys, can require buffering intermediate results and for large datasets this can require large amounts of memory.

In order to avoid allocating memory until the OS or the container system kills the process, DataFusion operators only allocate memory they are able to reserve from the configured MemoryPool. Once the memory tracked by the pool is exhausted, operators must either free memory by spilling to local disk or error.

A MemoryPool can be shared by concurrently executing plans in the same process to control memory usage in a multi-tenant system.

The following memory pool implementations are available:

Required Methods§

source

fn grow(&self, reservation: &MemoryReservation, additional: usize)

Infallibly grow the provided reservation by additional bytes

This must always succeed

source

fn shrink(&self, reservation: &MemoryReservation, shrink: usize)

Infallibly shrink the provided reservation by shrink bytes

source

fn try_grow( &self, reservation: &MemoryReservation, additional: usize ) -> Result<()>

Attempt to grow the provided reservation by additional bytes

On error the allocation will not be increased in size

source

fn reserved(&self) -> usize

Return the total amount of memory reserved

Provided Methods§

source

fn register(&self, _consumer: &MemoryConsumer)

Registers a new MemoryConsumer

Note: Subsequent calls to Self::grow must be made to reserve memory

source

fn unregister(&self, _consumer: &MemoryConsumer)

Records the destruction of a MemoryReservation with MemoryConsumer

Note: Prior calls to Self::shrink must be made to free any reserved memory

Implementors§