Skip to main content

write_segments_positional

Function write_segments_positional 

Source
pub fn write_segments_positional<D>(
    out: &File,
    n_segments: usize,
    offsets: &[u64],
    n_workers: usize,
    decode: D,
) -> WriteStats
where D: Fn(usize, &mut Vec<u8>) -> Result<(), String> + Sync,
Expand description

Write independent segments of one output stream at pre-computed offsets, across a no-barrier, self-dispatching worker pool — each worker decodes AND pwrites its own segment, with no ordered collector and no serial concatenation.

The caller has already laid out the stream: offsets[i] is the byte offset in out where segment i‘s decoded bytes belong (a prefix-sum of the segments’ known output sizes), and out has been pre-sized to the total (e.g. via set_len). Each worker claims the next segment index, calls decode(i, &mut buf) to fill its reused buffer, and write_all_at (pwrite) writes it at offsets[i] — positional writes to disjoint regions never conflict, so exact stream byte-order is preserved without any ordering on the critical path.

decode should verify the decoded length matches what the caller assumed for offsets (returning Err on mismatch); a failed segment is logged and skipped, and the caller can detect failed > 0 to fall back to a safe path.