bixverse-rs

Description

A Rust crate for computational biology: statistics, gene set enrichment, graph algorithms, matrix factorisation, ontologies and a full single cell suite. It is the numerical core underneath the bixverse R package, but it has no R dependency of its own and can be used as a plain Rust library.

The code originally lived in src/Rust/ of that R package. Pulling it out into its own crate means the numerics get tested, benchmarked and versioned independently of the wrapper that happens to call them. Design constraint throughout: atlas-scale analysis on a laptop, not on a cluster.

Crates

Since 0.8.0 this is a workspace of smaller crates, so a consumer pulls in the layer it needs and not the dependency graph of everything else. They share one version and are released together.

Crate	What it holds	Builds on
`bixverse-sparse`	The compressed sparse structures and their byte serialisation
`bixverse-core`	`core`, `utils`, `ml`: linear algebra, statistics, SIMD, k-means, Gaussian processes; re-exports the sparse structures	sparse
`bixverse-methods`	`enrichment`, `graph`, `methods`, `ontology`	core
`bixverse-sc-data`	The single cell data layer: the binary store, its readers and writers, MTX (plain or gzipped), h5ad / 10x h5 behind `h5`, and obs / var to Parquet behind `parquet`	sparse
`bixverse-sc`	The rest of the single cell suite	methods, sc-data
`bixverse-gpu`	GPU linear algebra and bulk methods via `cubecl`	methods
`bixverse-sc-gpu`	GPU single cell methods	sc, gpu
`bixverse-mlx`	Experimental MLX backend, Apple Silicon only	sc
`bixverse-rs`	Facade: everything above under the module paths the crate always had	all of them, optional
`bixverse-r`	The extendr conversions; re-exports `bixverse-rs`	the facade

Only bixverse-r links libR, and only bixverse-sc-data with h5 pulls in HDF5.

What's in it

Well over a hundred named methods, plus the shared numerical scaffolding they sit on.

Domain	Methods
`core`	Correlations and pairwise similarity, PCA/SVD, randomised SVD, LOESS, RBF kernels, mutual information and entropy, scalar optimisation (R's `optimize`), linear mixed models (REML with Satterthwaite), E-distance and perturbation distances, sparse structures, synthetic data generators
`enrichment`	GSEA (fgsea multi-level), blitzGSEA (gamma-approximated null), GSVA, ssGSEA, singscore, mitch, over-representation, ULM and MLM activity scores (decoupler)
`graph`	Leiden, Louvain, WalkTrap, spectral clustering, label propagation, personalised and constrained PageRank, connected components, Dijkstra (single and multi-source), Kruskal spanning forests, PAGA-style graph abstraction, graph metrics
`methods`	NMF (bulk, dense HALS, sparse HALS, consensus, refit), ICA, LDA via variational Bayes, sparse multiple CCA, differential correlation, graph diffusion, SNF, RBH, CoReMo, dgRDL, cis-target; bulk DGE via edgeR's quasi-likelihood chain and limma-voom (feature `dge`)
`ml`	k-means, clustering metrics, landmark Gaussian process regression, Matern kernels
`ontology`	GO Elim, Wang and Resnik-style semantic similarity
`single_cell`	I/O for h5ad and 10x h5 (parallel gzip chunk reads), mtx (plain or gzipped, one pass), R/Seurat counts and a versioned binary sparse format (single and multi-file, mmap-backed, mergeable); QC, CellSweep ambient removal, HVG, scTransform v2 (per-sample fits, covariate regression, analytic Pearson residuals, corrected counts), PCA, kNN, SNN, MAGIC; doublet detection (Scrublet, scDblFinder, cxds); batch correction (Harmony v1 and v2, BBKNN, fastMNN, Seurat CCA/rPCA anchors); integration metrics (kBET, iLISI/cLISI, silhouette, PC regression); annotation (scType, Symphony); DGE (Mann-Whitney with AUROC, pseudobulk edgeR, NEBULA); analysis (SCENIC with regulon binarisation, AUCell, Hotspot, VISION, ULM and MLM activity scores, DIALOGUE, MELD, miloR, NicheNet, NMF, module scoring, fast k-means/Louvain clustering); meta cells (SEACells, MetaCells2, SuperCell, hdWGCNA, ULM and MLM activity scores, plus density and compactness metrics); trajectories (Palantir, PAGA, diffusion maps, Markov chains, gene trends); Bonsai trees from Sanity posteriors (feature `bonsai`); multi-modal (WNN, DSB)
`gpu`	Sparse randomised SVD, SpMM and sparse GEMM, skinny GEMM, Gram, CholeskyQR2, correlation, NMF and consensus NMF; with `sc-gpu` also PCA, kNN, Harmony, BBKNN, SCENIC, Scrublet, SEACells, NEBULA, Sanity for Bonsai, fast clustering

Heavy lifting goes through faer for dense linear algebra (GEMM through Apple Accelerate on macOS), rayon for CPU fan-out, wide for SIMD, and cubecl for GPU kernels. Vector search, distance metrics and k-means come from the sister crate ann-search-rs; the GPU primitives layer from cubecl-utils-rs; the edgeR, limma and NEBULA numerics from edge-rs.

Feature flags

The flags below are the facade's (bixverse-rs); bixverse-r has the same ones. Each forwards to the crate that owns the dependency, so a consumer of a single crate sets that crate's flag instead (h5 on bixverse-sc-data, dge on bixverse-methods, and so on).

Flag	What it turns on
(always on)	Bulk statistics, enrichment, graph, ontology, matrix factorisation
`accelerate`	On by default. Dense CPU GEMM through Apple Accelerate on macOS, no effect elsewhere. See below
`dge`	Negative binomial DGE via `edge-rs`: bulk edgeR and limma-voom. Implied by `single-cell`, usable on its own
`single-cell`	The `single_cell` module, mmap I/O and the binary sparse format. No HDF5
`h5`	The h5ad and 10x h5 readers and writers. Pulls in HDF5, implies `single-cell`
`hdf5-static`	Builds the bundled HDF5 from source instead of linking an external `libhdf5`, implies `h5`
`hdf5-filters`	Compiles the LZF and Blosc filters into HDF5, so h5ad files written with those codecs load, implies `h5`
`parquet`	Writes the h5ad obs / var frames and the 10x barcodes and features to Parquet, ready for DuckDB's `read_parquet`. No arrow, implies `h5`
`multi-modal`	`single_cell::multi_modal` (WNN, DSB), implies `single-cell`
`bonsai`	Bonsai trees over Sanity posteriors via `bonsai-rs`, implies `single-cell`
`gpu`	The `gpu` module via `cubecl` (wgpu and CPU backends) and `cubek`: the bulk GPU methods
`sc-gpu`	`gpu::sc_gpu`, the GPU single cell methods, implies `gpu` and `single-cell`
`mlx`	Experimental, Apple Silicon only: an MLX backend via `mlx-c` (f32) for correlation, SpMM, randomised SVD, NMF (dense and sparse, consensus, k sweeps), single cell PCA, Harmony and SEACells. Independent of `gpu`, implies `single-cell`. Needs `brew install mlx-c` or `MLX_C_PREFIX`; without the library the build warns and skips it
`large-test`	Slow but asserting tests: GPU parity gates, large-scale numerical checks
`large_scale_diagnostics`	Unasserted diagnostic sweeps that print tables for a human

macOS and Apple Accelerate

On macOS the default accelerate feature sends every dense CPU GEMM (correlations, covariance, perturbation distances, NMF HALS, Hotspot, Markov chains, the landmark GP) through Apple's Accelerate framework. It ships with every macOS, so there is nothing to install. Turn it off with default-features = false. End to end on an M1 Max:

function	f32	f64
`column_pairwise_cov`, 50k x 2k	2.3x	1.8x
`column_pairwise_cor`, 50k x 2k	1.9x	1.7x
dense NMF HALS, 20k x 2k, k = 10	1.3x	1.6x
dense NMF HALS, 20k x 2k, k = 50	1.8x	1.8x

Using it from Rust

Everything through the facade:

[dependencies]
bixverse-rs = { version = "0.9", features = ["single-cell", "h5"] }

use bixverse_rs::prelude::*;

Or only the layer you need, e.g. the binary store without HDF5:

[dependencies]
bixverse-sc-data = "0.9"

Every crate has a prelude that re-exports the ones below it: the errors, sparse structures, SparseGraph, matrix and vector helpers, the SIMD trait and the assertion macros. Each crate has its own error enum (CoreErrors, MethodsErrors, ScDataErrors, ScErrors, ...), and each wraps the ones below it, so ? converts upwards.

How it plugs into R

The R side is a thin wrapper, and deliberately so. bixverse depends on this crate through extendr, and does the things R is good at: argument checking, S7 classes, documentation, plotting handoff. The Rust side owns the maths.

The R conversions live in their own crate, bixverse-r, which re-exports the facade, so an R package needs that one dependency. It holds the *_r_wrapper.rs files and three traits: FromRList deserialises a parameter struct from an R list, FromRMap does the same from an already flattened one, and ToRList serialises fitted structures back. The #[extendr] entry points themselves live downstream in the R packages, so R types never touch the numerical surface, and no crate other than bixverse-r links libR.

R packages that consume it:

bixverse with single-cell, h5, parquet and multi-modal. The main package.
bixverse.gpu with sc-gpu, adding wgpu-backed versions of the expensive methods. It does not need HDF5.
manifoldsR, on the core layer only.

Anything on the Rust side that never crosses into R works standalone, so the crate is equally usable from a pure Rust binary or behind a PyO3 layer.

Using it from Python

The bixverse Python package wraps the single cell streaming engine with PyO3 and maturin. It depends on bixverse-sc-data alone, so the wheel needs no R install. It opens a bixverse binary store and streams shuffled minibatches out of it, with the decode in Rust and the GIL released, fast enough to keep a GPU training loop fed. First consumer is scVI in bixverse.nn, on torch or MLX, trained straight off the store.

uv pip install bixverse            # numpy only
uv pip install "bixverse[torch]"   # scVI on torch
uv pip install "bixverse[mlx]"     # scVI on MLX, Apple Silicon

Docs live here. The Python package is versioned on its own line, see its changelog.

Roadmap

Methods

NMF for dense and sparse data
NicheNet for single cell, see Browaeys et al..
Palantir for single-cell trajectories, see Setty, et al..
Slingshot for single-cell trajectories, see Street, et al..

Differential expression

Bulk edgeR quasi-likelihood and limma-voom via edge-rs
Pseudobulk and NEBULA for single cell
limma treat/topTreat, decideTests, and block/correlation in voom

GPU accelerations

GPU-accelerated sparse, randomised SVD
GPU-accelerated Harmony
GPU-accelerated correlations (Metal does not have the absolute greatest performance here unfortunately...)
SEACells with GPU acceleration.
GPU-accelerated BBKNN version
GPU-accelerated NEBULA

Python interface

Data loader to pull the counts out of the binary files and expose them to Python-based deep learning frameworks (PyTorch, MLX).
scVI trained straight off the binary files, on torch and MLX.

Updates

Updates on what's happening in this crate can be found here

Licence

MIT License

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

bixverse-methods 0.9.4