Struct ParallelSparseReader 

pub struct ParallelSparseReader { /* private fields */ }

ParallelSparseReader

Implementations

impl ParallelSparseReader

pub fn new(f_path: &str) -> Result<Self, ScDataErrors>

Generate a new parallelised streaming reader

Params

• f_path - Path to the file

Returns

Initialised ParallelSparseReader.

pub fn file_version(&self) -> u32

Format version of the file this reader maps.

Returns

SC_FILE_VERSION_V3 or SC_FILE_VERSION.

pub fn count_norm_stored_chunks(&self) -> Result<usize, ScDataErrors>

Count the chunks that store their normalised layer verbatim.

Only reads the fixed chunk headers. Such chunks skip the norm recompute on read and are larger on disk.

Returns

The number of norm-stored chunks, or ScDataErrors::FileVersionMismatch on a v3 file, which always stores the layer.

impl ParallelSparseReader

pub fn read_cells_csr( &self, indices: &[usize], data_layer: &DataLayerReturn, ) -> Result<CsrBatch, ScDataErrors>

Read a batch of cells straight into CSR, as f32.

SingleCellReading::read_cells_sparse with T = f32, unpacked into the CsrBatch the Python loader hands to scipy or torch.

Params

• indices - Original cell indices, in the order the batch wants them
• data_layer - Which layer(s) to materialise

Returns

The batch as CSR, or ScDataErrors::ReaderModeMismatch on a gene-major store.

Trait Implementations

impl SingleCellReading for ParallelSparseReader

fn read_cells_parallel( &self, indices: &[usize], ) -> Result<Vec<CsrCellChunk>, ScDataErrors>

Read in cells by indices in a multi-threaded manner

Params

• indices - Slice of index positions of the cells to retrieve

Returns

Returns an array of CsrCellChunk.

fn read_cells_sparse<T>( &self, indices: &[usize], data_layer: &DataLayerReturn, ) -> Result<CompressedSparseData2<T, f32>, ScDataErrors>

where T: BixverseNumeric + FromPrimitive,

Read a batch of cells straight into CSR

Decompresses each cell into the worker’s scratch buffer and appends its payload in place, see [read_fused].

Params

• indices - Cell indices, in output row order
• data_layer - Which layer(s) to materialise

Returns

The batch as CSR.

fn read_genes_sparse<T>( &self, indices: &[usize], cells_to_keep: Option<&IndexSet<u32>>, data_layer: &DataLayerReturn, ) -> Result<CompressedSparseData2<T, f32>, ScDataErrors>

where T: BixverseNumeric + FromPrimitive,

Read a batch of genes straight into CSC, optionally filtering cells

Decompresses each gene into the worker’s scratch buffer and appends its payload in place, filtering through a dense cell map built once per call, see [read_fused].

Params

• indices - Gene indices, in output column order
• cells_to_keep - Cells to keep, in output row order, or None
• data_layer - Which layer(s) to materialise

Returns

The batch as CSC.

fn read_gene_parallel( &self, indices: &[usize], ) -> Result<Vec<CscGeneChunk>, ScDataErrors>

Read in genes by indices in a multi-threaded manner

Params

• indices - Slice of index positions of the genes to retrieve

Returns

Returns an array of CscGeneChunk.

fn read_gene_parallel_raw( &self, indices: &[usize], ) -> Result<Vec<CscGeneChunk>, ScDataErrors>

Read genes with raw counts and indices only.

On v4 the norm layer is neither recomputed nor converted, data_norm comes back empty and the decoded indices move into the chunk. v3 reads both layers, as Self::read_gene_parallel.

Params

• indices - Slice of index positions of the genes to retrieve

Returns

Returns an array of CscGeneChunk with an empty data_norm on v4.

fn read_gene_parallel_filtered( &self, indices: &[usize], cells_to_keep: &IndexSet<u32>, ) -> Result<Vec<CscGeneChunk>, ScDataErrors>

Read in genes by indices in a multi-threaded manner

And filter the genes by cells_to_keep in parallel. Overrides the default so that reading and filtering are fused into a single pass.

Params

• indices - Slice of index positions of the genes to retrieve
• cells_to_keep - Cell index positions to keep, in output order

Returns

Returns an array of CscGeneChunk.

fn get_header(&self) -> SparseDataHeader

Helper to return the header

Returns

Returns the header file

fn read_cell_library_sizes( &self, indices: &[usize], ) -> Result<Vec<usize>, ScDataErrors>

Read only library sizes for specified cells

More efficient than the default, which parses whole chunks. Files with a library-size index read 8 bytes per cell from it; older files decompress each chunk and read the header bytes (offset 12-19) without deserialising the rest.

Params

• indices - Cell indices

Returns

Vector of library sizes

fn read_gene_nnz(&self, indices: &[usize]) -> Result<Vec<usize>, ScDataErrors>

Read the NNZ for specific genes

Overrides the default, which parses whole chunks. Reads directly from the chunk header bytes (offset 16-23) instead.

Params

• indices - Gene indices

Returns

Vector of number of NNZ genes.

fn is_cell_based(&self) -> bool

Is the reader connected to a cell-based file?

Returns

Bool

fn target_size(&self) -> Option<f32>

Library size the data_norm layer of this file was scaled to.

Returns

Some(target_size), or None for raw-only files and for files written before the field entered the header.

fn is_gene_based(&self) -> bool

Is the store laid out for fast gene retrieval?

fn read_cell(&self, index: usize) -> Result<CsrCellChunk, ScDataErrors>

Read a single cell by index

fn read_gene(&self, index: usize) -> Result<CscGeneChunk, ScDataErrors>

Read a single gene by index

fn read_cells_range( &self, start: usize, end: usize, ) -> Result<Vec<CsrCellChunk>, ScDataErrors>

Read a contiguous range of cells

fn get_all_cells(&self) -> Result<Vec<CsrCellChunk>, ScDataErrors>

Return all cells in the store

fn get_all_genes(&self) -> Result<Vec<CscGeneChunk>, ScDataErrors>

Return all genes in the store

fn get_all_gene_nnz(&self) -> Result<Vec<usize>, ScDataErrors>

Return NNZ for all genes

fn get_clr_offsets( &self, indices: &[usize], batch_size: Option<usize>, ) -> Result<Vec<f64>, ScDataErrors>

Compute the shifted CLR per-cell offsets freshly from raw counts.