Crate oxinfer 

oxinfer: CPU-first inference for Gemma 4 and Qwen 3.5/3.8 in pure Rust, with optional runtime-detected accelerators.

Modules

audio

Gemma 4 audio: log-mel feature extraction (USM style), the conformer encoder with chunked local attention and relative position bias, and the projection into the text embedding space.

chat

Chat formatting for each model family (mirrors their chat templates), including tool calling.

clock

Monotonic time. std::time::Instant is unavailable on wasm32-unknown-unknown (it panics), so there the host supplies a millisecond clock through the host_now_ms import.

config

Gemma 4 text-model configuration, parsed from HuggingFace config.json.

cuda

CUDA backend (Linux / Windows, NVIDIA): the driver API and NVRTC are loaded at run time with dlsym, so the binary has no CUDA dependency and runs unchanged on machines without a GPU. The GEMM kernels are compiled from source by NVRTC for the detected compute capability.

drafter

Gemma 4 “assistant” drafter for speculative decoding: a 4-layer, 256-wide decoder whose layers all cross-attend to the main model’s shared KV caches. Input is the concatenation of the main model’s (scaled) embedding of the last token and the main model’s hidden state of the token before it; output is an argmax draft token (over a centroid-selected vocabulary subset) and a projected hidden state to feed the next draft step.

fsx

File access behind one small interface so the model loaders work both on a real filesystem and in a browser, where there is none: on wasm32-unknown-unknown the host (JavaScript) registers files in an in-memory table (vfs_put) and the loaders read them from there. Large files are stored page-aligned and handed out by move, so a checkpoint is never copied inside the module.

gpu

Accelerator backends: a trait the model’s linear layers dispatch through, plus a global registry chosen at start-up from the platform detection. Every backend is optional at run time; the CPU kernels remain the reference path and the fallback.

hf

Fetching and caching HuggingFace model repositories by name, over rsurl (pure-Rust HTTP).

kernels

CPU compute kernels: bf16 weights, f32 activations, multi-threaded via the spin pool.

media

Media decoding through oxideav (pure-Rust codecs): images to RGB8, audio to 16 kHz mono f32.

mm_config

Multimodal (vision / audio) configuration from config.json.

model

Gemma 4 text decoder: embeddings, per-layer embeddings (PLE), sliding/global attention with KV sharing, gated-GELU MLP, tied softcapped LM head.

platform

Runtime platform detection: CPU architecture and SIMD features, core count, memory, and GPUs (CUDA, Metal, Vulkan) probed by loading the vendor libraries at run time with dlsym, so the binary never links against them and runs everywhere with the CPU path as the fallback.

png

A dependency-free PNG writer for 8-bit RGB images (zlib “stored” blocks: no compression, so the file is about 3 bytes per pixel, but it works on every target, the browser included).

pool

Minimal spin-waiting thread pool for low-latency parallel-for.

prof

Lightweight always-on profiler: accumulated wall time per category.

quant

Int8 block quantization (Q8: blocks of 32 weights sharing one f32 scale).

qwen

Qwen3.5 / Qwen3.8 dense hybrid text model: Gated DeltaNet (linear attention with a causal short conv and a delta-rule recurrent state) interleaved with gated full attention, SiLU MLPs and zero-centered RMSNorm. Text only (the vision encoder and MTP head are not loaded).

safetensors

Minimal zero-copy safetensors reader (single file or sharded directory).

sampler

Token sampling: greedy, or temperature + top-k + top-p.

sd

Text-to-image with Stable Diffusion 1.x / 2.x and SD-Turbo from a diffusers folder (model_index.json, text_encoder/, tokenizer/, unet/, vae/, scheduler/).

tools

Tool (function) calling: tool declarations in each model family’s own syntax, parsing of the calls a model emits, and a few built-in tools for the command line. The host runs the tools; chat::ChatFormat feeds the results back in the format the model was trained on.

vision

Gemma 4 vision: image preprocessing (aspect-preserving bicubic resize into 16x16 patches), the ViT encoder with axial 2D RoPE and bidirectional attention, 3x3 patch pooling, and the projection of soft tokens into the language model’s embedding space.