Skip to main content

Module spill

Module spill 

Source
Expand description

Disk spilling for queries that exceed available memory.

When a sort or aggregation grows too large for RAM, we spill partitions to disk and merge them back later. This lets queries complete even with limited memory - just slower.

ComponentPurpose
SpillManagerManages spill file lifecycle with automatic cleanup
SpillFileRead/write individual spill files
ExternalSortExternal merge sort for big ORDER BY
PartitionedStateHash partitioning for spillable GROUP BY

Async variants (AsyncSpillManager, AsyncSpillFile) use tokio for non-blocking I/O when running in async contexts.

Structs§

AsyncSpillFile
Async handle for a single spill file.
AsyncSpillFileReader
Async reader for a spill file.
AsyncSpillManager
Async manager for spill file lifecycle for out-of-core processing.
ExternalSort
External merge sort for out-of-core sorting.
PartitionedState
Partitioned accumulator state for spillable aggregation.
SortKey
Sort key specification.
SpillFile
Handle for a single spill file.
SpillFileReader
Reader for a spill file.
SpillManager
Manages spill file lifecycle for out-of-core processing.

Enums§

NullOrder
Null handling in sort.
SortDirection
Sort direction.

Constants§

DEFAULT_NUM_PARTITIONS
Default number of partitions for hash partitioning.

Functions§

deserialize_row
Deserializes a row from bytes.
deserialize_value
Deserializes a Value from bytes.
serialize_row
Serializes a row (slice of Values) to bytes.
serialize_value
Serializes a Value to bytes.