Struct SparseMtxData 

pub struct SparseMtxData { /* private fields */ }

10x-like cell-feature matrix with zarr backend (feature x cell)

(root)
    ├── nrow
    ├── ncell
    ├── by_column
    │   ├── data
    │   ├── indices (row indices)
    │   └── indptr (column pointers)
    └── by_row
        ├── data
        ├── indices (column indices)
        └── indptr (row pointers)

Implementations

impl SparseMtxData

pub fn new(zarr_file: Option<&str>) -> Result<Self>

Create an empty new SparseMtxData instance with a zarr backend file If no backend_file is provided, a temporary file will be created.

• backend_file - Optional zarr backend file

pub fn open(backend_file: &str) -> Result<Self>

Create SparseMtxData instance from an existing zarr backend file

• zarr_file - zarr backend file (directory or .zarr.zip)

pub fn from_mtx_file( mtx_file: &str, backend_file: Option<&str>, index_by_row: Option<bool>, ) -> Result<Self>

Create SparseMtxData from mtx file with backend_file as the backend file. If no backend_file is provided, it will be the same as mtx_file with .zarr extension.

• mtx_file: mtx file to be read into zarr backend
• backend_file: zarr file to be associated with
• index_by_row: if true, the matrix will be indexed by row

pub fn from_ndarray( array: &Array2<f32>, zarr_file: Option<&str>, index_by_row: Option<bool>, ) -> Result<Self>

Create a new SparseMtxData instance from an ndarray array

• array - 2D array to be added to the backend
• backend_file - Optional zarr backend file
• index_by_row - Optional flag to index by row (CSR format)

pub fn from_dmatrix( matrix: &DMatrix<f32>, zarr_file: Option<&str>, index_by_row: Option<bool>, ) -> Result<Self>

Create a new SparseMtxData instance from an DMatrix array

• array - 2D array to be added to the backend
• backend_file - Optional zarr backend file
• index_by_row - Optional flag to index by row (CSR format)

pub fn print_hierarchy(&self) -> Result<()>

Show the hierarchy of the zarr store

Trait Implementations

impl Clone for SparseMtxData

fn clone(&self) -> Self

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl SparseIo for SparseMtxData

fn read_row_indptr(&mut self) -> Result<()>

Read row index pointers

fn column_indptr(&self) -> &[u64]

Read column index pointers

fn preload_columns(&mut self) -> Result<()>

preload columns’ values and indices

fn preload_rows(&mut self) -> Result<()>

preload rows’ values and indices

fn record_mtx_shape( &mut self, mtx_shape: Option<(usize, usize, usize)>, ) -> Result<()>

Helper function to keep the matrix shape

fn initialize_backend(&mut self) -> Result<()>

Helper function to create a new zarr backend file

fn remove_backend_file(&self) -> Result<()>

Clean up the backend file

fn get_backend_file_name(&self) -> &str

Access file name of the zarr backend

fn to_mtx_file(&self, mtx_file: &str) -> Result<()>

Export the data to a mtx file. This will take time.

• mtx_file: mtx file to be written

fn register_row_names_file(&mut self, row_name_file: &str)

Set row names for the matrix

• row_name_file: a file each line contains row name words

fn register_row_names_vec(&mut self, rows: &[Box<str>])

Set row names for the matrix

• rows: a vector of row names

fn register_column_names_file(&mut self, column_name_file: &str)

Set column names for the matrix

• column_name_file: a file each line contains column name words

fn register_column_names_vec(&mut self, columns: &[Box<str>])

Set column names for the matrix

• columns: a vector of column names

fn num_rows(&self) -> Option<usize>

Number of rows in the matrix

fn num_columns(&self) -> Option<usize>

Number of columns in the matrix

fn num_non_zeros(&self) -> Option<usize>

Number of non-zero elements in the matrix

fn register_names_file( &mut self, key: &str, name_file: &str, name_columns: Range<usize>, name_sep: &str, ) -> Result<()>

Add arbitrary names (a vector of strings)

• group_name: group name
• name_file: a file each line contains name words
• name_columns: range of columns to be used for name
• name_sep: separator for name columns

fn register_names_vec(&mut self, key: &str, names: &[Box<str>]) -> Result<()>

Add arbitrary names (a vector of strings)

• group_name: group name
• names: a file each line contains name words

fn retrieve_registered_names(&self, key: &str) -> Result<Vec<Box<str>>>

Get back the registered names

• key: key for the registered names

fn read_triplets_by_single_column( &self, j_data: usize, ) -> Result<(usize, usize, Vec<(u64, u64, f32)>)>

Read columns within the range and return a vector of triplets (row, col, value)

• col : usize

fn read_triplets_by_columns( &self, columns: Self::IndexIter, ) -> Result<(usize, usize, Vec<(u64, u64, f32)>)>

Read columns within the range and return dense ndarray::Array2

• columns : range e.g., 0..3 -> [0, 1, 2] or vec![0, 1, 2]

fn read_triplets_by_rows( &self, rows: Self::IndexIter, ) -> Result<(usize, usize, Vec<(u64, u64, f32)>)>

Read rows within the range and return a vector of triplets (row, col, value)

• rows : range e.g., 0..3 -> [0, 1, 2] or vec![0, 1, 2]

fn record_csr_dataset_backend( &mut self, csr_cols: &[u64], csr_vals: &[f32], csr_rowptr: &[u64], ) -> Result<()>

CSR data structure in Zarr backend

    └── by_row
        ├── data
        ├── indices (column indices)
        └── isndptr (row pointers)

fn record_csc_dataset_backend( &mut self, csc_rows: &[u64], csc_vals: &[f32], csc_colptr: &[u64], ) -> Result<()>

CSC data structure in Zarr backend

Helper function to record the CSC dataset
    ├── by_column
    │   ├── data
    │   ├── indices (row indices)
    │   └── indptr (column pointers)

type IndexIter = Vec<usize>

fn row_indptr(&self) -> &[u64]

The resident by-row indptr; the row-major twin of column_indptr, empty when absent.

fn metadata(&self) -> Metadata

The strings stored with the data to say what it is (see meta for the keys programs share); empty when none were set.

fn set_metadata(&mut self, values: &Metadata) -> Result<()>

Store meta as the data’s metadata, replacing what was there. It needs a writable backend: one being created (set it before zipping), or an unzipped zarr store. A zipped zarr store, or an HDF5 file opened to read, refuses it.

fn reopen_backend(&mut self) -> Result<()>

Re-open handles on the CURRENT backend path after its contents were replaced from outside (a finished temp file renamed into place). The zarr store is path-addressed so this is a cache refresh; hdf5 holds an open file handle that would otherwise point at the deleted inode.

fn read_column_indptr(&mut self) -> Result<()>

preload column index pointers

fn clean_preloaded_columns(&mut self)

unload the memory

fn clean_preloaded_rows(&mut self)

unload the row memory

fn backend_type(&self) -> SparseIoBackend

backend file type

fn row_names(&self) -> Result<Vec<Box<str>>>

fn column_names(&self) -> Result<Vec<Box<str>>>

fn csc_column_arrays(&self) -> Option<(&[u64], &[u64], &[f32])>

Zero-copy view of preloaded column-major CSC arrays as (indptr, indices, data). Returns None when the backend has not preloaded columns or doesn’t support direct array access. Callers (e.g. SparseIoVec::read_columns_csc) use this to skip the triplet roundtrip when columns are already in memory.

fn cs_create(&mut self, key: CsKey, len: usize) -> Result<()>

Create a fixed-size 1-D backend dataset of len elements for the given CSC/CSR slot. No data is written yet.

fn cs_write_u64(&mut self, key: CsKey, offset: u64, data: &[u64]) -> Result<()>

Write a u64 slab at offset in the specified dataset. Used for CSC/CSR indices and indptr.

fn cs_write_f32(&mut self, key: CsKey, offset: u64, data: &[f32]) -> Result<()>

Write an f32 slab at offset in the specified dataset. Used for CSC/CSR data.

fn read_columns_ndarray(&self, columns: Self::IndexIter) -> Result<Array2<f32>>

Read columns within the range and return dense ndarray::Array2

fn read_columns_tensor(&self, columns: Self::IndexIter) -> Result<Tensor>

Read columns within the range and return dense candle_core::Tensor

fn read_columns_dmatrix(&self, columns: Self::IndexIter) -> Result<DMatrix<f32>>

Read columns within the range and return dense nalgebrea::DMatrix

fn read_columns_csr(&self, columns: Self::IndexIter) -> Result<CsrMatrix<f32>>

Read columns within the range and return sparse CsrMatrix

fn read_columns_csc(&self, columns: Self::IndexIter) -> Result<CscMatrix<f32>>

Read columns within the range and return sparse CsrMatrix

fn read_rows_ndarray(&self, rows: Self::IndexIter) -> Result<Array2<f32>>

Read rows within the range and return dense ndarray::Array2

fn read_rows_tensor(&self, rows: Self::IndexIter) -> Result<Tensor>

Read rows within the range and return dense candle_core::Tensor

fn read_rows_dmatrix(&self, rows: Self::IndexIter) -> Result<DMatrix<f32>>

Read rows within the range and return dense nalgebra::DMatrix

fn read_rows_csr(&self, rows: Self::IndexIter) -> Result<CsrMatrix<f32>>

Read rows within the range and return sparse CsrMatrix

fn read_rows_csc(&self, rows: Self::IndexIter) -> Result<CscMatrix<f32>>

Read rows within the range and return sparse CscMatrix

fn import_mtx_file(&mut self, mtx_file: &str, index_by_row: bool) -> Result<()>

Read an mtx file once and populate the backend: the column (CSC) index always, the row (CSR) index as well when index_by_row. Both are streamed out of the same triplet vector, so the file is inflated once and the triplets are the only full-size structure alive.

fn import_dmatrix_by_row(&mut self, matrix: &DMatrix<f32>) -> Result<()>

Add dmatrix to zarr backend by row (CSR format)

fn import_dmatrix_by_col(&mut self, matrix: &DMatrix<f32>) -> Result<()>

Add dmatrix to zarr backend by column (CSC format)

fn import_ndarray_by_row(&mut self, array: &Array2<f32>) -> Result<()>

Add ndarray to zarr backend by row (CSR format)

fn import_ndarray_by_col(&mut self, array: &Array2<f32>) -> Result<()>

Add ndarray to zarr backend by column (CSC format)

fn column_nnz(&self, col: usize) -> Option<u64>

Exact nnz of one column, from the resident indptr — no I/O.

fn meta(&self, key: &str) -> Option<String>

One metadata value, e.g. meta(meta::SAMPLE).

fn set_meta(&mut self, key: &str, value: &str) -> Result<()>

Set one metadata value, keeping the others.

fn subset_columns_rows( &mut self, columns: Option<&Vec<usize>>, rows: Option<&Vec<usize>>, ) -> Result<()>

Keep the columns columns and the rows rows of the data, each in the order listed (None keeps that side whole), replacing the backend’s content. The result is streamed into a sibling and swapped in once complete, so a refused selection leaves the data as it was.

fn reorder_rows(&mut self, row_names_order: &[Box<str>]) -> Result<()>

Reposition rows in the order of row_names_order: row i of the result is the row named row_names_order[i], all-zero when the data has no such row; rows whose name is not listed are dropped. The result is streamed into a sibling and swapped in once complete, like subset_columns_rows.

fn record_triplets_by_row( &mut self, row_col_val_triplets: &mut Vec<(u64, u64, f32)>, ) -> Result<()>

Stream the triplets out as CSR slabs; the row-major twin of record_triplets_by_col.

fn record_triplets_by_col( &mut self, row_col_val_triplets: &mut Vec<(u64, u64, f32)>, ) -> Result<()>

Stream the triplets out as CSC slabs.

fn begin_streaming_csc(&mut self, shape: (usize, usize, usize)) -> Result<()>

Begin a streaming CSC build for a sparse matrix of known shape. Pre-creates /by_column/{data, indices, indptr} at their final sizes so subsequent append_csc_slab calls write into disjoint hyperslabs without further allocation.

fn append_csc_slab( &mut self, col_offset: u64, nnz_offset: u64, local_colptr: &[u64], row_indices: &[u64], values: &[f32], ) -> Result<()>

Append one contiguous CSC column band.

fn finalize_streaming_csc(&mut self) -> Result<()>

Finalize CSC streaming by writing the final indptr sentinel at position ncol, equal to the total nnz.

fn begin_streaming_csr(&mut self, shape: (usize, usize, usize)) -> Result<()>

Begin a streaming CSR build for a sparse matrix of known shape; the row-major twin of begin_streaming_csc.

fn append_csr_slab( &mut self, row_offset: u64, nnz_offset: u64, local_rowptr: &[u64], col_indices: &[u64], values: &[f32], ) -> Result<()>

Append one contiguous CSR row band; the row-major twin of append_csc_slab, with the same audits.

fn finalize_streaming_csr(&mut self) -> Result<()>

Finalize CSR streaming: write the indptr sentinel at position nrow, load the row index, and check the appended count against the declared nnz – the one violation the written indptr cannot reveal.

fn build_csr_from_csc_streaming(&mut self) -> Result<()>

Build /by_row/{data, indices, indptr} by transposing the already- written CSC data on disk: one pass over the CSC to count each row, then one scatter pass per row band. The band is as large as the process-wide preload budget and the memory available now allow (never under SLAB_BUDGET_BYTES), so a matrix that fits is transposed in a single scatter pass.