Skip to main content

Module parallel

Module parallel 

Source
Expand description

Phase 15.4: the bounded worker pool for parallel ingest. Only meaningful with the non-default parallel feature (which implies field); without it the module is a zero-sized marker and every ingest runs serially. Bounded CPU parallelism for the naturally independent ingest work (Phase 15.4).

The pool is compiled only with the non-default parallel feature (which implies field). Without that feature WorkerPool is a zero-sized marker that is never constructed, and every ingest site takes its serial path; the exact core therefore keeps no new dependency and no behavior change.

§The one architectural rule

A worker may only run a pure map over immutable inputs. It must never touch a FieldStore, its Rc-based IoCounters, a derived-cache memo, or the disk. The store’s single-threaded accounting is the fuse that keeps parallelism honest: the parallel phase computes owned canonical bytes and NodeIds, and a serial fold merges them in physical order.

Determinism is therefore structural, not a tuning property. Indexed par_iter().collect() preserves physical order, and every order-dependent gate (the running MAX_TOTAL_DECODED / MAX_OBJSTM* caps, the existence probes and counters) is replayed in that same serial fold — so the parallel path yields byte-identical nodes, ids, index entries, and limits enforcement to the serial path.

Structs§

WorkerPool
Without the parallel feature this is a zero-sized marker that is never constructed; the ingest sites keep their serial path (pool is always None).