Skip to main content

stream_column_selection

Function stream_column_selection 

Source
pub fn stream_column_selection(
    data: &dyn SparseIo<IndexIter = Vec<usize>>,
    selected_columns: &[usize],
    row_filter: Option<&[usize]>,
    out_row_names: &[Box<str>],
    out_col_names: &[Box<str>],
    file_out: &str,
    backend_out: &SparseIoBackend,
) -> Result<(usize, usize, usize)>
Expand description

Stream a column selection (with an optional ASCENDING row filter) into a fresh backend, without ever materialising the survivors.

The shape split and subsample share: pick columns, optionally keep an ascending subset of rows, write a new file. Both used to read every surviving triplet into one Vec and hand it to the sorting triplet writer — 24 B/nnz of residency that OOMs at imaging scale. Here the exact output nnz comes from the resident indptr (or one counting pass when rows are filtered), and survivors flow through the validated streaming writer in byte-budgeted slabs.

row_filter, when given, must be ascending: the renumbering is then monotone, so within-column row order survives and no per-column sort is needed. Both callers sample or select ascending; the debug assert keeps the next caller honest rather than silently writing unsorted columns (which the streaming writer would refuse anyway).