Module parallel_scan 

Parallel scan coordinator — Fase 4 P5 building block.

Generic worker-pool helper for chunked collection scans. Mirrors PG’s nodeGather.c + execParallel.c at a much smaller scale: splits an input into equal chunks, processes each chunk on a separate OS thread, and funnels results back to the caller in a single ordered Vec.

The module intentionally doesn’t depend on rayon — reddb avoids large transitive dep graphs and rayon is overkill for the single-query-level parallelism we need here. A plain std::thread::spawn + mpsc::channel approach gives us enough throughput for the scan sizes that matter (1k-10m rows per query).

Usage pattern:

let out = parallel_scan(
    &records,
    num_cpus(),
    |chunk| chunk.iter().filter(|r| matches(r)).cloned().collect(),
);

The closure runs on each chunk; its output is a Vec that the coordinator concatenates in chunk order, preserving input ordering for the merged result.

This module is not yet wired into any executor. The full-scan path in runtime/query_exec/table.rs is the obvious first call site once the planner learns to flag scans as “parallel-eligible” (size > threshold, no side-effecting filters).

Parallelism decisions

Parallelism overhead dominates at small sizes — thread spawn + channel setup is ~1 µs per thread. The coordinator falls back to sequential execution when:

• chunk_count == 1 (no parallelism available)
• Input length < min_parallel_rows (tuned constant, default 4096)

Above the threshold the input is sliced into roughly chunk_count equal pieces and each piece ships to a worker thread. The main thread joins and concatenates.

Structs

ScanBatches

Lazily-evaluated, pull-based counterpart to parallel_scan.

Constants

DEFAULT_SCAN_BATCH_ROWS

Default number of rows processed per pulled batch. Sized so the per-batch working set stays well under one engine page worth of row payload for typical row widths, while amortising the per-batch closure-call overhead. The streaming chunk producer (S1) re-buffers these into page-aligned wire chunks downstream, so this is purely the executor’s internal pull granularity.

MIN_PARALLEL_ROWS

Minimum input length below which parallel execution falls back to sequential. Tuned so the overhead of thread spawn

Functions

default_parallelism

Number of worker threads to use by default. Currently clamped to num_cpus().min(8) — more than 8 workers for a single query tends to thrash the buffer pool.

parallel_count

Count-only variant of parallel_scan that avoids materialising the intermediate Vec. The worker returns a usize (number of matching rows in its chunk), and the coordinator sums them.

parallel_count_default

Phase 3.4 wiring entry point for COUNT(*) over a filtered scan. Same default parallelism as parallel_scan_default.

parallel_scan

Execute worker across input in parallel using chunk_count threads. Preserves input order in the output: chunk 0’s results come first, chunk 1’s next, etc.

parallel_scan_default

Phase 3.4 wiring entry point. Calls parallel_scan with default_parallelism() worker count. Used by the runtime scan executor when its planner cost model decides parallel is profitable (input size > MIN_PARALLEL_ROWS). Saves the caller from manually threading the worker count through.

parallel_scan_rows

Per-row flattening helper over parallel_scan_stream. Yields one U at a time — the natural shape for a record-at-a-time streaming driver — while keeping the same lazy, bounded-memory pull semantics (at most one batch is materialised at a time).

parallel_scan_stream

Construct a pull-based ScanBatches over input, applying worker to each batch_rows-sized slice on demand. A batch_rows of 0 is clamped to 1 so the iterator always makes progress.