Trait datafusion::execution::memory_pool::MemoryPool

source ·

pub trait MemoryPool: Send + Sync + Debug {
    // Required methods
    fn grow(&self, reservation: &MemoryReservation, additional: usize);
    fn shrink(&self, reservation: &MemoryReservation, shrink: usize);
    fn try_grow(
        &self,
        reservation: &MemoryReservation,
        additional: usize
    ) -> Result<(), DataFusionError>;
    fn reserved(&self) -> usize;

    // Provided methods
    fn register(&self, _consumer: &MemoryConsumer) { ... }
    fn unregister(&self, _consumer: &MemoryConsumer) { ... }
}

Expand description

The pool of memory on which MemoryReservations record their memory reservations.

DataFusion is a streaming query engine, processing most queries without buffering the entire input. However, certain operations such as sorting and grouping/joining with a large number of distinct groups/keys, can require buffering intermediate results and for large datasets this can require large amounts of memory.

In order to avoid allocating memory until the OS or the container system kills the process, DataFusion operators only allocate memory they are able to reserve from the configured MemoryPool. Once the memory tracked by the pool is exhausted, operators must either free memory by spilling to local disk or error.

A MemoryPool can be shared by concurrently executing plans in the same process to control memory usage in a multi-tenant system.

The following memory pool implementations are available: