Struct TrackConsumersPool 

pub struct TrackConsumersPool<I> { /* private fields */ }

A MemoryPool that tracks the consumers that have reserved memory within the inner memory pool.

By tracking memory reservations more carefully this pool can provide better error messages on the largest memory users when memory allocation fails.

Tracking is per hashed MemoryConsumer, not per MemoryReservation. The same consumer can have multiple reservations.

Automatic Usage via RuntimeEnvBuilder

The easiest way to use TrackConsumersPool is via RuntimeEnvBuilder::with_memory_limit().

Usage Examples

For more examples of using TrackConsumersPool, see the memory_pool_tracking.rs example

Implementations

impl<I: MemoryPool> TrackConsumersPool<I>

pub fn new(inner: I, top: NonZeroUsize) -> Self

Creates a new TrackConsumersPool.

Arguments

• inner - The underlying memory pool that handles actual memory allocation
• top - The number of top memory consumers to include in error messages

Note

In most cases, you should use RuntimeEnvBuilder::with_memory_limit() instead of creating this pool manually, as it automatically sets up tracking with sensible defaults (top 5 consumers).

Example

use std::num::NonZeroUsize;
use datafusion_execution::memory_pool::{TrackConsumersPool, GreedyMemoryPool, FairSpillPool};

// Create with a greedy pool backend, reporting top 3 consumers in error messages
let tracked_greedy = TrackConsumersPool::new(
    GreedyMemoryPool::new(1024 * 1024), // 1MB limit
    NonZeroUsize::new(3).unwrap(),
);

// Create with a fair spill pool backend, reporting top 5 consumers in error messages
let tracked_fair = TrackConsumersPool::new(
    FairSpillPool::new(2 * 1024 * 1024), // 2MB limit
    NonZeroUsize::new(5).unwrap(),
);

Impact on Error Messages

The top determines how many Top K MemoryConsumers to include in the reported DataFusionError::ResourcesExhausted.

pub fn report_top(&self, top: usize) -> String

Returns a formatted string with the top memory consumers.

Trait Implementations

impl<I: Debug> Debug for TrackConsumersPool<I>

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl<I: MemoryPool> MemoryPool for TrackConsumersPool<I>

fn register(&self, consumer: &MemoryConsumer)

Registers a new MemoryConsumer

fn unregister(&self, consumer: &MemoryConsumer)

Records the destruction of a MemoryReservation with MemoryConsumer

fn grow(&self, reservation: &MemoryReservation, additional: usize)

Infallibly grow the provided reservation by additional bytes

fn shrink(&self, reservation: &MemoryReservation, shrink: usize)

Infallibly shrink the provided reservation by shrink bytes

fn try_grow( &self, reservation: &MemoryReservation, additional: usize, ) -> Result<()>

Attempt to grow the provided reservation by additional bytes

fn reserved(&self) -> usize

Return the total amount of memory reserved

fn memory_limit(&self) -> MemoryLimit

Return the memory limit of the pool