Enum EngineMemory 

pub enum EngineMemory {
    Unbounded,
    Private(usize),
    Shared {
        pool: Arc<dyn MemoryPool>,
        fair_share_bytes: usize,
    },
}

Where an engine’s DataFusion execution memory comes from.

The distinction that matters is Private versus Shared. An executor runs slots task fragments at once; giving each its own pool means the process claims slots × pool_size, which is how a container gets OOM-killed while every individual pool still reports headroom. One pool shared by every task makes the executor’s total execution memory a single hard number, and FairSpillPool divides it live across whoever is actually running — a task alone gets all of it, four tasks get a quarter each, and nobody has to predict the concurrency in advance.

Variants

Unbounded

DataFusion’s default unbounded pool: no accounting, no spill.

Private(usize)

A FairSpillPool of this size belonging to this engine alone. Correct for one-engine-per-process deployments (embedded, gateway).

Shared

A pool shared with every other engine in this process.

fair_share_bytes is what this engine can expect when all slots are busy. It sizes spill reservations only — the pool itself is the shared object and is not bounded by this number.

Fields

pool: Arc<dyn MemoryPool>

The process-wide pool.

fair_share_bytes: usize

This engine’s expected share, for reservation sizing.

Implementations

impl EngineMemory

pub fn from_limit(bytes: Option<usize>) -> Self

A private pool of bytes, or Unbounded when there is no limit.

pub fn shared_pool(bytes: usize) -> Arc<dyn MemoryPool>

Build a pool of bytes intended to be shared by several engines.

The caller keeps the returned handle and passes clones of it in EngineMemory::Shared, which is what bounds their combined execution memory rather than each engine’s individually.

pub fn for_this_process() -> Self

The default memory source for an engine built in this process: a share of process_query_pool.

This is what makes the bound hold no matter how many engines a process ends up with. A process can host the Flight SQL engine, several executor task slots, and IVM tick engines at once; when each built its own pool at a fraction of the container, the fractions summed past the container and the container was OOM-killed while every pool still reported headroom. One shared pool cannot oversubscribe, and FairSpillPool divides it live — an engine running alone gets all of it, so the bound costs nothing when there is no contention.

pub fn sizing_bytes(&self) -> Option<usize>

The byte figure that should size spill reservations and session config, or None when execution memory is unbounded.

Trait Implementations

impl Clone for EngineMemory

fn clone(&self) -> EngineMemory

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for EngineMemory

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.