Skip to main content

Module io

Module io 

Source
Expand description

I/O and partition helpers used by the generated engine code.

  • partition — split an owned Vec into per-worker slices for the library-mode batch engine’s ingest path.
  • byte_range_reader — split a CSV file across timely workers so each reads its own byte slice (binary mode).
  • shard_int / shard_str / shard_spur — pick the owning worker for a tuple based on its first column (binary mode).

§Byte-range reader example

if let Some((reader, budget)) = byte_range_reader(path, index, peers) {
    let mut buf = Vec::new();
    let mut consumed = 0u64;
    while consumed < budget {
        buf.clear();
        let n = reader.read_until(b'\n', &mut buf).unwrap_or(0);
        if n == 0 { break; }
        consumed += n as u64;
        // parse &buf …
    }
}

Functions§

byte_range_reader
Open a byte-range slice of path for worker index out of peers.
partition
Split v into n roughly-equal owned partitions, in order.
shard_int
Shard an integer-typed first column across peers workers.
shard_spur
Shard an interned-string first column (lasso::Spur) across peers.
shard_str
Shard a string-typed first column across peers workers.