pub fn load_input(paths: &[PathBuf], memory_limit: u64) -> Result<(Input, u64)>Expand description
Loads paths into memory when their total size is at most half of
memory_limit, leaving room for the compressed copy; otherwise the input
is streamed from the files.
Also returns the input’s size, measured here once: the bytes read into memory, or the files’ sizes when streamed. It is the ratio’s denominator, and each file’s length is snapshotted with it (reads stop there), so a file still growing while the rows run changes neither.
A file whose size cannot be read, or that cannot be opened, is a usage error naming it (exit 2); only a failing read is an i/o error.