kjarni 0.8.0

Transformer inference library: run embeddings, classification, reranking, search, chat, speech-to-text, summarization and translation inside your program. Same output as PyTorch.
Documentation

Kjarni

Run transformer models inside your program, in any language. Same output as PyTorch.

Kjarni is a transformer inference library written in Rust: embeddings, classification, reranking, search, chat, speech to text, summarization and translation, on CPU or GPU inside your process. This crate is its Rust API; the same engine ships for Python, C#, Go, C++ and the browser.

[dependencies]
kjarni = "0.2"
tokio = { version = "1", features = ["macros", "rt-multi-thread"] }
use kjarni::Embedder;

let embedder = Embedder::new("minilm-l6-v2").await?;
println!("{}", embedder.similarity("doctor", "physician").await?);  // 0.8598
println!("{}", embedder.similarity("doctor", "banana").await?);     // 0.3379

Models load from HuggingFace safetensors and GGUF directly. The first call downloads and caches under ~/.cache/kjarni; nothing leaves the machine.

What it implements

BERT-style encoders, cross-encoders, Llama-family decoders (Llama, Qwen2, Mistral, Phi-3), T5, BART and Whisper. Quantised weights in Q4_K, Q6_K, Q8_0 and Q8_K, on CPU with AVX2/FMA and NEON kernels or on GPU through WebGPU, which means no CUDA toolkit.

For an arbitrary research model, convert it and use ONNX Runtime. For embeddings, classification, reranking or chat inside a program that has to ship somewhere, this is one dependency with no toolchain.

The workspace

Crate What it is
kjarni The public API. Start here.
kjarni-transformers The engine: kernels, model graphs, CPU and GPU backends
kjarni-models Model definitions and the registry
kjarni-search Vector search primitives
kjarni-rag Retrieval-augmented generation
kjarni-cli The kjarni command line tool
kjarni-ffi The C ABI every other language binding is built on

How fast it is

An i7-13700 and an RTX A2000 12 GB, each engine in its own process. On this NVIDIA card, engines using CUDA are well ahead of Kjarni's WebGPU; what WebGPU gives you is the same code on NVIDIA, AMD, Intel and Apple hardware with nothing to install.

Embeddings per second on the CPU

Embeddings per second on the GPU

Speech to text speed on CPU and GPU

Text generation on the CPU

Text generation on the GPU

These are the Rust engine's own numbers measured through its bindings; the Rust API and the C ABI take the same time, 3.43 and 3.44 ms for one short MiniLM embedding. Method and every number: BENCHMARKS.md.

The same engine elsewhere

One engine, one set of kernels, the same numbers from every language.

License

MIT or Apache-2.0.