Skip to main content

Module parallel_build

Module parallel_build 

Source
Expand description

Parallel cold build of the derived read-model (ADR-107/ADR-108).

Native shape of services/parallel_build.py + parallel_merge.py: the per-document work (parse, classify, index row, token vectors, live/scope projection) fans out across rayon workers as compact fragments; the parent merge runs only the cross-document steps (graph resolution, inbound fill, portfolio) — in sorted-path order, so the store bytes are worker-count-invariant by construction (no pickling boundary exists in-process; ADR-114).

Correctness never depends on the parallel rung: below the file-count threshold, on a 1–2 core box, or on ANY worker fault (a panic in a fragment task — exercised by DECIDED_PARALLEL_BUILD_FAULT), the build falls back to the authoritative serial floor, whose partial results are never written.

Structs§

BuildStats
Per-phase cold-build timings for the DECIDED_TIMING scorecard line.

Constants§

DEFAULT_MIN_PARALLEL_FILES
Below this file count the fan-out’s coordination overhead outweighs the win, so the cold build stays on the serial floor (the oracle’s measured crossover, kept for contract fidelity).

Functions§

build_derived_index_parallel
Build the derived read-model with a parallel parse AND per-doc derive — byte-identical to derived::build_derived_index for any worker count.
emit_build_timing
Write the cold-build scorecard line to stderr when DECIDED_TIMING is set — env-gated, stderr-only, byte-shaped like the oracle’s (ADR-107).
parallel_parse_paths
The paths type alias for the freshness tracker’s explicit-list parse (INDEX-PLAN B6): parse a known path list through the one true per-file path, parallel when it pays; entries in list order.