Module spill 

Transparent spilling for out-of-core query processing.

This module provides infrastructure for spilling operator state to disk when memory pressure is high, enabling queries to complete even when intermediate results exceed available memory.

Architecture

• SpillManager - Manages spill file lifecycle with automatic cleanup (sync)
• AsyncSpillManager - Async version using tokio for non-blocking I/O
• SpillFile - Read/write abstraction for individual spill files (sync)
• AsyncSpillFile - Async version using tokio
• Serializer functions for binary Value encoding (no serde overhead)
• ExternalSort - External merge sort for out-of-core sorting
• PartitionedState - Hash partitioning for spillable aggregation

Structs

AsyncSpillFile

Async handle for a single spill file.

AsyncSpillFileReader

Async reader for a spill file.

AsyncSpillManager

Async manager for spill file lifecycle for out-of-core processing.

ExternalSort

External merge sort for out-of-core sorting.

PartitionedState

Partitioned accumulator state for spillable aggregation.

SortKey

Sort key specification.

SpillFile

Handle for a single spill file.

SpillFileReader

Reader for a spill file.

SpillManager

Manages spill file lifecycle for out-of-core processing.

Enums

NullOrder

Null handling in sort.

SortDirection

Sort direction.

Constants

DEFAULT_NUM_PARTITIONS

Default number of partitions for hash partitioning.

Functions

deserialize_row

Deserializes a row from bytes.

deserialize_value

Deserializes a Value from bytes.

serialize_row

Serializes a row (slice of Values) to bytes.

serialize_value

Serializes a Value to bytes.