Struct SparseIoVec 

pub struct SparseIoVec { /* private fields */ }

Implementations

impl SparseIoVec

pub fn register_batches_ndarray<T>( &mut self, feature_matrix: &Array2<f32>, batch_membership: &[T], ) -> Result<()>

where T: Sync + Send + Hash + Eq + Clone + ToString,

Register batch membership information along with the feature matrix for quick look up operations.

Arguments

• feature_matrix - A feature matrix where each column corresponds to a cell.
• batch_membership - A vector of batch membership information for each cell.

pub fn register_batches_dmatrix<T>( &mut self, feature_matrix: &DMatrix<f32>, batch_membership: &[T], ) -> Result<()>

where T: Sync + Send + Hash + Eq + Clone + ToString,

Register batch membership information along with the feature matrix for quick look up operations.

Arguments

• feature_matrix - A feature matrix where each column corresponds to a cell.
• batch_membership - A vector of batch membership information for each cell.

pub fn batch_name_map(&self) -> Option<HashMap<Box<str>, usize>>

pub fn num_batches(&self) -> usize

pub fn batch_knn_lookup(&self) -> Option<&Vec<ColumnDict<usize>>>

Borrow the per-batch HNSW lookups populated by build_hnsw_per_batch / register_batches_dmatrix. Returns None before the indices have been built.

pub fn register_batch_membership<T>(&mut self, batch_membership: &[T])

where T: Sync + Send + Hash + Eq + Clone + ToString,

Register batch membership information without building HNSW indices. This is a lightweight alternative to register_batches_dmatrix for use with pb-sample based batch correction.

pub fn register_column_multiplicity( &mut self, multiplicity: &[f32], ) -> Result<()>

Declare that each column stands for more than one observation.

A column is normally one cell, so every statistic that divides by a count adds 1 per column. That breaks when a column is a summary of many cells — a carried pseudobulk, or a bulk sample — because the per-cell rate μ = Σy / n would divide a whole group’s counts by one.

With multiplicities registered, a column holding the mean profile of m cells and a weight of m contributes exactly what those m cells would have: m·mean to the sums and m to the count.

Absent (the default) every column weighs 1, and every accumulation is bit-for-bit what it was before this existed.

Errors

If multiplicity is not one entry per column, or holds a non-finite or non-positive weight — a zero would silently delete a column from the denominator while leaving its counts in the numerator.

pub fn column_multiplicity(&self, col: usize) -> f32

Weight of a single column — 1.0 when no multiplicities are registered.

pub fn has_column_multiplicity(&self) -> bool

True when any column stands for more than one observation.

pub fn column_multiplicities(&self) -> Option<&[f32]>

The whole multiplicity vector, one weight per column — None when no multiplicities are registered (every column is one observation).

Prefer this over gathering Self::column_multiplicity in a loop: callers were rebuilding the vector element-by-element, re-encoding the “absent means 1.0” default at every site.

pub fn batch_names(&self) -> Option<Vec<Box<str>>>

pub fn batch_to_columns(&self, batch: usize) -> Option<&Vec<usize>>

pub fn get_batch_membership<I>(&self, cells: I) -> Vec<usize>

where I: Iterator<Item = usize>,

pub fn column_names(&self) -> Result<Vec<Box<str>>>

impl SparseIoVec

pub fn assign_groups<T>( &mut self, column_to_group: &[T], ncolumns_per_group: Option<usize>, )

where T: Sync + Send + Hash + Eq + Clone + ToString,

Assign columns to groups

• column_to_group - column to group membership
• ncolumns_per_group - number of columns per group. None: assign all the columns to the groups; Some(x): limit the maximum number of columns per group to at most x.

pub fn take_grouped_columns(&self) -> Option<&Vec<Vec<usize>>>

Take a vector of columns where each vector corresponds to a set

pub fn group_keys(&self) -> Option<&Vec<Box<str>>>

Get the group keys in the same order as group indices

pub fn group_key_to_cols(&self) -> Option<HashMap<Box<str>, Vec<usize>>>

Get a mapping from group keys to their column indices

pub fn take_backend_columns(&self) -> Vec<(Box<str>, Vec<usize>)>

Take a vector of backend file and corresponding column indices

pub fn get_group_membership<I>(&self, cells: I) -> Result<Vec<usize>>

where I: Iterator<Item = usize>,

Recall the cells group assignment; Note that this can be differ from the original vector used in assign_groups as we can have different number of columns and groups.

pub fn num_groups(&self) -> usize

number of groups

impl SparseIoVec

pub fn read_neighbouring_columns_csc<I>( &self, cells: I, knn_batches: usize, knn_columns: usize, skip_same_batch: bool, skip_batches: Option<&[usize]>, ) -> Result<(CscMatrix<f32>, Vec<usize>, Vec<usize>, Vec<f32>)>

where I: Iterator<Item = usize>,

Take columns within the neighbourhood of given cells

Arguments

• cells - global column indices
• target_batches - the batches for targeted kNN search
• knn_batches - k-nearest neighbour batches
• knn_columns - k-nearest neighbour columns
• skip_same_batch - skip the same batch

Returns

• the knn-matched matrix
• source_columns - a vector of the source columns
• a vector of distances between the matched columns

pub fn read_neighbouring_columns_ndarray<I>( &self, cells: I, knn_batches: usize, knn_columns: usize, skip_same_batch: bool, skip_batches: Option<&[usize]>, ) -> Result<(Array2<f32>, Vec<usize>, Vec<f32>)>

where I: Iterator<Item = usize>,

Take columns neighbouring with the given cells

Arguments

• cells - global column indices
• target_batches - the batches for targeted kNN search
• knn - k-nearest neighbours
• skip_same_batch - skip the same batch

Returns

• the knn-neighbouring matrix
• source_columns - a vector of the source columns
• distances - a vector of distances between the neighbouring columns

pub fn read_neighbouring_columns_dmatrix<I>( &self, cells: I, knn_batches: usize, knn_columns: usize, skip_same_batch: bool, skip_batches: Option<&[usize]>, ) -> Result<(DMatrix<f32>, Vec<usize>, Vec<f32>)>

where I: Iterator<Item = usize>,

Take columns neighbouring with the given cells

Arguments

• cells - global column indices
• target_batches - the batches for targeted kNN search
• knn - k-nearest neighbours
• skip_same_batch - skip the same batch

Returns

• the knn-neighbouring matrix
• source_columns - a vector of the source columns
• distances - a vector of distances between the neighbouring columns

pub fn read_matched_columns_csc<I>( &self, cells: I, target_batches: &[usize], knn: usize, skip_same_batch: bool, ) -> Result<(CscMatrix<f32>, Vec<usize>, Vec<f32>)>

where I: Iterator<Item = usize>,

Take columns matched with the given cells

Arguments

• cells - global column indices
• target_batches - the batches for targeted kNN search
• knn - k-nearest neighbours
• skip_same_batch - skip the same batch

Returns

• the knn-matched matrix
• source_columns - a vector of the source columns
• a vector of distances between the matched columns

pub fn read_matched_columns_ndarray<I>( &self, cells: I, target_batches: &[usize], knn: usize, skip_same_batch: bool, ) -> Result<(Array2<f32>, Vec<usize>, Vec<f32>)>

where I: Iterator<Item = usize>,

Take columns matched with the given cells

Arguments

• cells - global column indices
• target_batches - the batches for targeted kNN search
• knn - k-nearest neighbours
• skip_same_batch - skip the same batch

Returns

• the knn-matched matrix
• source_columns - a vector of the source columns
• distances - a vector of distances between the matched columns

pub fn read_matched_columns_dmatrix<I>( &self, cells: I, target_batches: &[usize], knn: usize, skip_same_batch: bool, ) -> Result<(DMatrix<f32>, Vec<usize>, Vec<f32>)>

where I: Iterator<Item = usize>,

Take columns matched with the given cells

Arguments

• cells - global column indices
• target_batches - the batches for targeted kNN search
• knn - k-nearest neighbours
• skip_same_batch - skip the same batch

Returns

• the knn-matched matrix
• source_columns - a vector of the source columns
• distances - a vector of distances between the matched columns

pub fn query_columns_by_data_csc<T>( &self, query: T, knn_per_batch: usize, ) -> Result<(CscMatrix<f32>, Vec<usize>, Vec<f32>)>

where T: MakeVecPoint,

Query columns with projection data

Arguments

• query - a data vector
• knn_per_batch - k-nearest neighbour columns per batch

Returns

• the knn-matched matrix
• source_columns - a vector of the source columns
• a vector of distances between the matched columns

pub fn query_columns_by_data_ndarray<T>( &self, query: T, knn_per_batch: usize, ) -> Result<(Array2<f32>, Vec<usize>, Vec<f32>)>

where T: MakeVecPoint,

Query columns with projection data

Arguments

• query - a data vector
• knn_per_batch - k-nearest neighbour columns per batch

Returns

• the knn-matched matrix
• source_columns - a vector of the source columns
• a vector of distances between the matched columns

pub fn query_columns_by_data_dmatrix<T>( &self, query: T, knn_per_batch: usize, ) -> Result<(DMatrix<f32>, Vec<usize>, Vec<f32>)>

where T: MakeVecPoint,

Query columns with projection data

Arguments

• query - a data vector
• knn_per_batch - k-nearest neighbour columns per batch

Returns

• the knn-matched matrix
• source_columns - a vector of the source columns
• a vector of distances between the matched columns

impl SparseIoVec

pub fn push( &mut self, data: Arc<dyn SparseIo<IndexIter = Vec<usize>>>, data_name: Option<Box<str>>, ) -> Result<()>

Add a backend’s columns to the vector.

• data: Arc to the backend [SparseData].
• data_name: under ColumnAlignment::Disjoint, appended as the @<data_name> display disambiguator for barcodes shared across files. Ignored under ColumnAlignment::Union, where cells glue by raw barcode — use push_with_barcode_suffix to attach a per-cell sample tag that participates in the Union merge.

pub fn push_with_barcode_suffix( &mut self, data: Arc<dyn SparseIo<IndexIter = Vec<usize>>>, data_name: Option<Box<str>>, barcode_suffix: Option<&str>, ) -> Result<()>

Like push but, under ColumnAlignment::Union, tags every barcode of this backend with {COLUMN_SEP}{barcode_suffix} before the canonical-merge step. Two backends that share a barcode merge into one global cell only if they carry the SAME suffix, so callers can encode per-file sample identity (e.g. rep1_wt): same-sample modalities merge, different samples stay distinct. The tagged name also becomes the displayed column name. None reproduces push exactly. The data_name Disjoint disambiguator is orthogonal: it still applies under Disjoint, and barcode_suffix only under Union (the two alignments are mutually exclusive).

pub fn num_columns_by_data(&self) -> Result<Vec<usize>>

pub fn remove_backend_file(&mut self) -> Result<()>

impl SparseIoVec

pub fn for_each_triplet<I, F>( &self, cells: I, chunk_cols: usize, f: F, ) -> Result<(usize, usize)>

where I: Iterator<Item = usize>, F: FnMut(u64, u64, f32),

Stream every nonzero of the selected cells (global column indices) through f(row, col, val) without materializing the full triplet vector. row is a compact-row index and col runs over 0..ncol in the iteration order of cells — exactly the indices Self::columns_triplets would emit.

Columns are processed in slabs of at most chunk_cols, so the only transient allocation is one backend slab’s worth of (u64, u64, f32) triplets: peak memory is bounded by chunk_cols, not by the total nnz. Callers can therefore build a compact edge list (e.g. 12-byte triplets) directly and never pay for the wide intermediate. Returns the (nrow, ncol) dimensions.

pub fn columns_triplets<I>( &self, cells: I, ) -> Result<((usize, usize), Vec<(u64, u64, f32)>)>

where I: Iterator<Item = usize>,

Collect all nonzeros of the selected cells into one triplet vector. Thin wrapper over Self::for_each_triplet with a single slab spanning every column (identical one-pass behavior); prefer for_each_triplet when the result is consumed once, to avoid the full-width intermediate.

pub fn read_columns_ndarray<I>(&self, cells: I) -> Result<Array2<f32>>

where I: Iterator<Item = usize>,

pub fn read_columns_dmatrix<I>(&self, cells: I) -> Result<DMatrix<f32>>

where I: Iterator<Item = usize>,

pub fn read_columns_csc<I>(&self, cells: I) -> Result<CscMatrix<f32>>

where I: Iterator<Item = usize>,

Direct-slice CSC read: bypasses the triplet → COO → CSC roundtrip when the underlying backends have preloaded column arrays.

For each cell, we slice (indices, values) straight out of the backend’s preloaded by_column_indices / by_column_data, remap row indices through l2g then g2c once, drop entries that fall outside the row intersection, and assemble final CSC arrays in one pass per column. Backends that aren’t preloaded fall back to per-column triplet reads, which still avoids the global triplet vec and the column-major sort inside CscMatrix::from(&coo).

pub fn read_columns_csr<I>(&self, cells: I) -> Result<CsrMatrix<f32>>

where I: Iterator<Item = usize>,

pub fn read_columns_tensor<I>(&self, cells: I) -> Result<Tensor>

where I: Iterator<Item = usize>,

pub fn rows_triplets<I>( &self, rows: I, ) -> Result<((usize, usize), Vec<(u64, u64, f32)>)>

where I: Iterator<Item = usize>,

Build (shape, triplets) for the requested compact rows across all backends. Output column index is the SparseIoVec-global column (concatenation of backends in push order); output row index is the position in rows.

pub fn read_rows_ndarray<I>(&self, rows: I) -> Result<Array2<f32>>

where I: Iterator<Item = usize>,

pub fn read_rows_dmatrix<I>(&self, rows: I) -> Result<DMatrix<f32>>

where I: Iterator<Item = usize>,

pub fn read_rows_csc<I>(&self, rows: I) -> Result<CscMatrix<f32>>

where I: Iterator<Item = usize>,

pub fn read_rows_csr<I>(&self, rows: I) -> Result<CsrMatrix<f32>>

where I: Iterator<Item = usize>,

pub fn read_rows_tensor<I>(&self, rows: I) -> Result<Tensor>

where I: Iterator<Item = usize>,

impl SparseIoVec

pub fn new() -> Self

an empty sparse io vector for horizontal data integration

pub fn with_row_alignment(self, mode: RowAlignment) -> Result<Self>

Switch row-name alignment between intersection (default) and union. Must be called BEFORE any push — the row-mapping recompute uses the current value of row_alignment. Errors if any backend has already been added.

pub fn with_row_canonicalizer( self, canon: impl Fn(&str) -> Box<str> + Send + Sync + 'static, ) -> Result<Self>

Install a row-name canonicalizer for fuzzy cross-backend row alignment. Must be called BEFORE the first push — returns an error if any backend has already been added (the existing row_names_by_global would otherwise mix raw and canonicalized forms).

Typical use: pass a GeneIndexResolver-style canonicalizer so ENSG00000000003_TSPAN6 (file A) and TSPAN6 (file B) collapse to a single row, instead of being silently dropped from the shared-row intersection.

pub fn with_per_backend_row_suffix(self, suffix: Vec<Box<str>>) -> Result<Self>

Install a per-backend feature-name suffix (one entry per backend, in push order). Each backend b’s rows are renamed {canon(row)}/{suffix[b]}, so files sharing raw feature names stay on separate rows unless they also share the suffix (same modality). Must be called BEFORE the first push. The vec length must match the number of backends that will be pushed; push errors if didx is out of range.

pub fn with_column_alignment(self, mode: ColumnAlignment) -> Result<Self>

Switch column (cell) alignment between disjoint concatenation (default) and barcode-keyed union. Must be called BEFORE any push — the push branches off the current value. Errors if any backend has already been added.

pub fn with_column_canonicalizer( self, canon: impl Fn(&str) -> Box<str> + Send + Sync + 'static, ) -> Result<Self>

Install a column-name canonicalizer for fuzzy cross-backend barcode matching under ColumnAlignment::Union. Must be called BEFORE the first push — returns an error if any backend has already been added. Has no effect under ColumnAlignment::Disjoint (barcodes are never compared across backends in that mode).

pub fn len(&self) -> usize

number of data sets

pub fn is_empty(&self) -> bool

check if the vector is empty

pub fn num_rows(&self) -> usize

pub fn num_rows_in_at_least(&self, k: usize) -> usize

Number of canonical rows observed by at least k of the pushed backends. Useful for detecting multi-modal-shaped inputs (num_rows_in_at_least(n_backends) is the strict intersection size; comparing it against per-backend row counts reveals how disjoint the feature axes are).

pub fn column_alignment(&self) -> ColumnAlignment

Current column-alignment mode. Mirrors Self::row_alignment.

pub fn row_coverage_by_backend(&self) -> Option<Vec<Vec<bool>>>

Per-backend row coverage on the exposed (compact) row axis: coverage[d][r] is true when backend d measures row r.

Under RowAlignment::Union a backend with a smaller panel simply has no entry at the rows it lacks — reads return zero there, which is indistinguishable from “measured, and absent”. This is the map that lets a consumer tell the two apart: unmeasured is no evidence, not evidence of zero.

None when every backend covers every row (single backend, identical panels, or intersect alignment) — the common case, so callers can skip observability handling entirely on None.

pub fn column_source(&self, col: usize) -> Option<usize>

The single backend a global column comes from, or None when the column merges several backends (column-union alignment). Observability accounting needs one source per column; a merged column has a set of panels, which callers must handle (or refuse) explicitly.

pub fn column_locations(&self, col: usize) -> &[BackendLocation]

Every backend location of global column col: one entry under Disjoint, one per observing backend under Union. Empty when out of range.

pub fn num_non_zeros(&self) -> Result<usize>

pub fn num_columns(&self) -> usize

total number of columns across all data files

pub fn clone_for_collapse(&self) -> Self

Structural clone for an independent projection / collapse pass.

Named entry point for the collapse copy; it is exactly self.clone(), kept as a method so call sites read as intent. Cloning copies the data handles (matrices are Arc-shared, so only the index Vecs/HashMaps are duplicated — cheap) and the row/column alignment state, while dropping every derived batch / group / HNSW cache — see [DerivedCaches], whose drop-on-clone behavior is what isolates the non-Clone batch_knn_lookup HNSW indices and lets SparseIoVec #[derive(Clone)] at all. Those caches are re-registered from scratch by register_batch_membership / the collapse, so a fresh clone is the correct pre-collapse state.

Prefer this name over a bare .clone() at collapse sites; note that any .clone() is likewise lossy on the derived caches.

Intended use: let mut spliced = vec.clone_for_collapse(); spliced.mask_rows(&spliced_keep)?; — gives a spliced-only view that drives RP + collapse + refinement without disturbing the full backend (still needed at all rows for per-modality aggregation).

pub fn mask_rows(&mut self, keep: &[bool]) -> Result<()>

Exclude rows (genes) from the working set. keep[compact_row] is true for rows to keep, false for rows to exclude. The compact row indices are renumbered after filtering. This affects all downstream operations (projection, collapse, training, inference).

pub fn mask_columns(&mut self, keep: &[bool]) -> Result<()>

Exclude columns (cells) from the working set. keep[global_col] is true for cells to keep, false to exclude. Global column indices are renumbered after filtering. This affects all downstream operations (projection, collapse, training, inference).

Cell-axis mirror of Self::mask_rows. MUST be called before batch/group registration: register_batch_membership and group assignment index by global column id and would be corrupted by a renumber, so we debug_assert! they are unset and defensively clear them. Dropped cells leave their backend-local columns in place but unmapped (usize::MAX in data_to_cols), which the row-wise read path (rows_triplets) skips.

pub fn clear_column_membership(&mut self)

Drop the cell-indexed group/batch membership caches so the columns can be re-masked and the membership re-registered from scratch. Needed before Self::mask_columns when a collapse already registered groups on this backend (it asserts the caches are unset); the next assign_groups / register_batch_membership / collapse rebuilds them. This is the fourth reset site alluded to in [DerivedCaches].

pub fn row_names(&self) -> Result<Vec<Box<str>>>

Trait Implementations

impl Clone for SparseIoVec

fn clone(&self) -> Self

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Default for SparseIoVec

fn default() -> Self

an empty sparse io vector for horizontal data integration

impl Index<usize> for SparseIoVec

type Output = Arc<dyn SparseIo<IndexIter = Vec<usize>>>

The returned type after indexing.

fn index(&self, idx: usize) -> &Self::Output

Performs the indexing (container[index]) operation.

impl VisitColumnsOps for SparseIoVec

fn visit_columns_by_block<Visitor, SharedIn, SharedOut>( &self, visitor: &Visitor, shared_in: &SharedIn, shared_out: &mut SharedOut, block_size: Option<usize>, ) -> Result<()>

where Visitor: Fn((usize, usize), &Self, &SharedIn, Arc<Mutex<&mut SharedOut>>) -> Result<()> + Sync + Send, SharedIn: Sync + Send + ?Sized, SharedOut: Sync + Send,

visit all the columns by sequential blocks. The visitor function should take (a) (lb, ub) (b) &Self (c) &SharedIn (d) Arc::new(Mutex::new(&mut SharedOut)

fn visit_columns_by_group<Visitor, SharedIn, SharedOut>( &self, visitor: &Visitor, shared_in: &SharedIn, shared_out: &mut SharedOut, ) -> Result<()>

where Visitor: Fn(usize, &[usize], &Self, &SharedIn, Arc<Mutex<&mut SharedOut>>) -> Result<()> + Sync + Send, SharedIn: Sync + Send + ?Sized, SharedOut: Sync + Send,

visit all the columns by predefined groups assigned by self.assign_groups. The visitor function should take (a) group_index (b) &[columns_in_the_group] (c) &Self (d) &SharedIn (e) Arc::new(Mutex::new(&mut SharedOut)