Expand description
Disk spilling for queries that exceed available memory.
When a sort or aggregation grows too large for RAM, we spill partitions to disk and merge them back later. This lets queries complete even with limited memory - just slower.
| Component | Purpose |
|---|---|
SpillManager | Manages spill file lifecycle with automatic cleanup |
SpillFile | Read/write individual spill files |
ExternalSort | External merge sort for big ORDER BY |
PartitionedState | Hash partitioning for spillable GROUP BY |
Async variants (AsyncSpillManager, AsyncSpillFile) use tokio for
non-blocking I/O when running in async contexts.
Structs§
- Async
Spill File - Async handle for a single spill file.
- Async
Spill File Reader - Async reader for a spill file.
- Async
Spill Manager - Async manager for spill file lifecycle for out-of-core processing.
- External
Sort - External merge sort for out-of-core sorting.
- Partitioned
State - Partitioned accumulator state for spillable aggregation.
- SortKey
- Sort key specification.
- Spill
File - Handle for a single spill file.
- Spill
File Reader - Reader for a spill file.
- Spill
Manager - Manages spill file lifecycle for out-of-core processing.
Enums§
- Null
Order - Null handling in sort.
- Sort
Direction - Sort direction.
Constants§
- DEFAULT_
NUM_ PARTITIONS - Default number of partitions for hash partitioning.
Functions§
- deserialize_
row - Deserializes a row from bytes.
- deserialize_
value - Deserializes a Value from bytes.
- serialize_
row - Serializes a row (slice of Values) to bytes.
- serialize_
value - Serializes a Value to bytes.