Module parallel_scan 

Parallel scan coordinator — Fase 4 P5 building block.

Generic worker-pool helper for chunked collection scans. Mirrors PG’s nodeGather.c + execParallel.c at a much smaller scale: splits an input into equal chunks, processes each chunk on a separate OS thread, and funnels results back to the caller in a single ordered Vec.

The module intentionally doesn’t depend on rayon — reddb avoids large transitive dep graphs and rayon is overkill for the single-query-level parallelism we need here. A plain std::thread::spawn + mpsc::channel approach gives us enough throughput for the scan sizes that matter (1k-10m rows per query).

Usage pattern:

let out = parallel_scan(
    &records,
    num_cpus(),
    |chunk| chunk.iter().filter(|r| matches(r)).cloned().collect(),
);

The closure runs on each chunk; its output is a Vec that the coordinator concatenates in chunk order, preserving input ordering for the merged result.

This module is not yet wired into any executor. The full-scan path in runtime/query_exec/table.rs is the obvious first call site once the planner learns to flag scans as “parallel-eligible” (size > threshold, no side-effecting filters).

Parallelism decisions

Parallelism overhead dominates at small sizes — thread spawn + channel setup is ~1 µs per thread. The coordinator falls back to sequential execution when:

• chunk_count == 1 (no parallelism available)
• Input length < min_parallel_rows (tuned constant, default 4096)

Above the threshold the input is sliced into roughly chunk_count equal pieces and each piece ships to a worker thread. The main thread joins and concatenates.

Constants

MIN_PARALLEL_ROWS

Minimum input length below which parallel execution falls back to sequential. Tuned so the overhead of thread spawn

Functions

default_parallelism

Number of worker threads to use by default. Currently clamped to num_cpus().min(8) — more than 8 workers for a single query tends to thrash the buffer pool.

parallel_count

Count-only variant of parallel_scan that avoids materialising the intermediate Vec. The worker returns a usize (number of matching rows in its chunk), and the coordinator sums them.

parallel_count_default

Phase 3.4 wiring entry point for COUNT(*) over a filtered scan. Same default parallelism as parallel_scan_default.

parallel_scan

Execute worker across input in parallel using chunk_count threads. Preserves input order in the output: chunk 0’s results come first, chunk 1’s next, etc.

parallel_scan_default

Phase 3.4 wiring entry point. Calls parallel_scan with default_parallelism() worker count. Used by the runtime scan executor when its planner cost model decides parallel is profitable (input size > MIN_PARALLEL_ROWS). Saves the caller from manually threading the worker count through.