pub fn convert_parallel(
db: &Database,
sink: &mut impl GameSink,
threads: usize,
batch: u32,
) -> Result<ConvertStats>Expand description
Converts the database into sink across threads workers, delivering the
games in game-number order whatever the thread count.
The id space is cut into batch-record chunks (DEFAULT_BATCH, 8,192),
every worker decodes a whole chunk into its own buffers, and one writer
stage — the calling thread — delivers the chunks in id order. The sink is
therefore called from one thread at a time, in exactly the sequence
for_each_game calls it in, with exactly the same payloads; the only
thing the thread count changes is how long it takes.
At most one wave of chunks is in flight, one per worker, and the next wave is decoded while the current one is written. Peak memory is therefore a few tens of MiB for a whole-database conversion at any thread count, and does not grow with the size of the database.
threads of 0 or 1 runs for_each_game instead: there is nothing to
overlap, and the sequential path is the one whose order everything else is
compared against.