Expand description
I/O and partition helpers used by the generated engine code.
partition— split an ownedVecinto per-worker slices for the library-mode batch engine’s ingest path.byte_range_reader— split a CSV file across timely workers so each reads its own byte slice (binary mode).shard_int/shard_str/shard_spur— pick the owning worker for a tuple based on its first column (binary mode).
§Byte-range reader example
ⓘ
if let Some((reader, budget)) = byte_range_reader(path, index, peers) {
let mut buf = Vec::new();
let mut consumed = 0u64;
while consumed < budget {
buf.clear();
let n = reader.read_until(b'\n', &mut buf).unwrap_or(0);
if n == 0 { break; }
consumed += n as u64;
// parse &buf …
}
}Functions§
- byte_
range_ reader - Open a byte-range slice of
pathfor workerindexout ofpeers. - partition
- Split
vintonroughly-equal owned partitions, in order. - shard_
int - Shard an integer-typed first column across
peersworkers. - shard_
spur - Shard an interned-string first column (
lasso::Spur) acrosspeers. - shard_
str - Shard a string-typed first column across
peersworkers.