⚡ nuvec

MLX for Rust, written in Rust. Lazy arrays, composable transforms and Metal kernels for Apple Silicon, with no bindings to MLX or any C/C++ library.

Documentation · Getting started · Benchmarks

nuvec brings MLX's programming model to Rust: arrays that are evaluated lazily on unified memory, grad/vjp/jvp/vmap/compile that compose freely, and the Metal kernels that make MLX fast, ported and tested against MLX itself. On top sits an LLM runtime that loads mlx-community checkpoints and generates as fast as mlx-lm.

🚧 Pre-alpha. The API can still change between releases. Everything listed below works today and is covered by tests.

✨ Highlights

• 🦀 Pure Rust. No C++ build, no FFI to MLX. Metal is reached through Apple's system frameworks (objc2-metal); unsafe stays in the Metal runtime (and one CPU matrix-product call), and every other crate forbids it.
• 💤 Lazy arrays on unified memory. Operations build a graph; nothing runs until you read a result. The CPU and GPU share memory, so there are no copies between them.
• 🔁 Composable transforms. grad, value_and_grad, vjp, jvp, vmap, custom_function, checkpoint and compile nest in any order, through complex numbers too.
• 🔥 MLX's Metal kernels. GEMM and GEMV, quantized products (1–8 bit), fused attention, RMSNorm, RoPE, sort, scan, mixture-of-experts gather kernels, plus your own kernels from source (fast::MetalKernel).
• 🧪 Tested against MLX. Nearly a thousand cases replay results recorded from real MLX, with random layouts, gradients and every dtype.
• 🤖 LLM runtime. Llama, Mistral and Qwen 2/3/3.5, dense, hybrid (Gated DeltaNet) or mixture-of-experts, straight from mlx-community or Hugging Face checkpoints; speculative decoding included.
• 💾 Zero-copy weights. safetensors and npy files are memory-mapped and handed to the GPU without a copy.

📊 Performance

Same machine, same day, same checkpoints: nuvec-lm against mlx-lm 0.32.2 (Apple M4 Pro, greedy decoding, medians of four runs; ratios above 1 favour nuvec).

Speculative decoding with prompt-lookup drafts doubles decode speed on text that repeats its input (Qwen3-8B: 53 → 106 tok/s), at about the same speed otherwise. Full tables and methodology: benchmarks.

🚀 Quick start

cargo add nuvec            # arrays, transforms, Metal kernels
cargo add nuvec-lm         # the LLM runtime (optional)

Arrays are lazy, and gradients are arrays too:

use nuvec::{Array, Result, grad, multiply, sum};

fn main() -> Result<()> {
    let x = Array::from_slice(&[1.0f32, 2.0, 3.0], &[3])?;
    let y = (&x * 2.0 + 1.0) * &x;              // builds a graph, computes nothing
    println!("{:?}", y.to_vec::<f32>()?);       // [3.0, 10.0, 21.0], on the GPU

    let f = |x: &Array| sum(&multiply(x, x)?, &[], false); // sum(x²)
    println!("{:?}", grad(f)(&x)?.to_vec::<f32>()?);       // [2.0, 4.0, 6.0]
    Ok(())
}

Generate text with an mlx-community checkpoint:

use nuvec_lm::{GenerateOptions, generate, load};

fn main() -> nuvec::Result<()> {
    let (model, tokenizer) = load("mlx-community/Qwen3-8B-4bit")?; // from the HF cache
    let prompt = tokenizer.encode("The capital of France is");
    let options = GenerateOptions { max_tokens: 32, ..Default::default() };
    let ids: Vec<u32> = generate(&model, &prompt, options)?.collect::<nuvec::Result<_>>()?;
    println!("{}", tokenizer.decode(&ids));
    Ok(())
}

Or from the command line (the model must be in the local Hugging Face cache, e.g. hf download mlx-community/Qwen3-8B-4bit):

cargo run --release -p nuvec-lm --example generate -- \
    --model mlx-community/Qwen3-8B-4bit --prompt "Write a haiku about Rust" --chat

📦 Crates

🖥️ Platforms

🛠️ Building from source

git clone https://github.com/hupe1980/nuvec && cd nuvec
just ci        # everything CI runs: fmt, clippy, tests, docs, Miri, cargo-deny, kernel checks

Without the Metal toolchain the build still succeeds: the shader sources are embedded and compiled at runtime (NUVEC_METALLIB=off selects this mode explicitly).

🧪 Against MLX

Python is not needed to build or test: the fixtures recorded from MLX are checked in. With uv installed you can re-record them and compare speed on your own machine:

NUVEC_PROFILE=1 prints the GPU time per kernel of any run (see profiling).

📄 License

Licensed under either of Apache License, Version 2.0 or MIT license, at your option. Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in this work, as defined in the Apache-2.0 license, shall be dual licensed as above, without any additional terms or conditions.

Ported MLX kernels keep their MIT attribution. nuvec is not affiliated with Apple.