Crate oxinfer 

oxinfer: CPU-first inference for Gemma 4 and Qwen 3.5/3.8 in pure Rust, with optional runtime-detected accelerators.

Re-exports

pub use oxideav_core;

Modules

audio

Gemma 4 audio: log-mel feature extraction (USM style), the conformer encoder with chunked local attention and relative position bias, and the projection into the text embedding space.

av

oxideav-core types as oxinfer’s in-memory media (the oxideav feature): decoded pictures, audio and video as VideoFrame / AudioFrame plus the CodecParameters that describe them, converted to what the towers take (RGB8, mono f32 at the model’s rate), and RGB output (Stable Diffusion) back to a frame.

bert

BERT-style encoders (BERT, RoBERTa, XLM-RoBERTa: BGE-M3 and friends): learned absolute positions, post-LayerNorm blocks, biased projections, exact (erf) GELU, bidirectional attention.

chat

Chat formatting for each model family (mirrors their chat templates), including tool calling.

clock

Monotonic time. std::time::Instant is unavailable on wasm32-unknown-unknown (it panics), so there the host supplies a millisecond clock through the host_now_ms import.

config

Gemma 4 / Gemma 3 text-model configuration, parsed from HuggingFace config.json.

cuda

CUDA backend (Linux / Windows, NVIDIA): the driver API and NVRTC are loaded at run time with dlsym, so the binary has no CUDA dependency and runs unchanged on machines without a GPU. The GEMM kernels are compiled from source by NVRTC for the detected compute capability.

decide

Typed decisions (“System 1” of the Jev models, e.g. autotrust/JEV-9B, autotrust/JEV-27B): a question about a piece of text answered with calibrated probabilities from one prefill pass, with no generation.

drafter

Gemma 4 “assistant” drafter for speculative decoding: a 4-layer, 256-wide decoder whose layers all cross-attend to the main model’s shared KV caches. Input is the concatenation of the main model’s (scaled) embedding of the last token and the main model’s hidden state of the token before it; output is an argmax draft token (over a centroid-selected vocabulary subset) and a projected hidden state to feed the next draft step.

embed

Text embeddings for sentence-transformers style models: a backbone (Gemma 3 / EmbeddingGemma, Qwen3 / Qwen3-Embedding, BERT-family encoders such as BGE-M3) followed by the modules listed in modules.json: pooling, dense projections, normalization. Prompts (query, document, …) come from config_sentence_transformers.json. BGE-M3’s sparse (lexical weights) and multi-vector (ColBERT) heads are loaded when their files are present.

engine

High-level text generation: load a model by path or repo id, hold a conversation with it, stream the reply token by token, cancel, and read statistics. This is what the CLI’s text modes do, as a library (the CLI is built from the same pieces: crate::generate, resolve_model, load_tokenizer, crate::mm).

fsx

File access behind one small interface so the model loaders work both on a real filesystem and in a browser, where there is none: on wasm32-unknown-unknown the host (JavaScript) registers files in an in-memory table (vfs_put) and the loaders read them from there. Large files are stored page-aligned and handed out by move, so a checkpoint is never copied inside the module.

generate

Text generation building blocks shared by the CLI and crate::engine: the common surface of the model families (TextLm), chunked prefill, the decoding loops (plain sampling, Qwen MTP and Gemma 4 drafter speculative decoding) with a per-token callback that can stop them, and incremental detokenization for streaming (TextStream).

ggml

llama.cpp / ggml weight formats, as stored in GGUF files: block layouts, exact dequantization and the decode dot products, so GGUF checkpoints run without re-quantizing.

gguf

GGUF (llama.cpp) checkpoints: the header (metadata and tensor table), the mapping from llama.cpp tensor names back to the Hugging Face names the loaders use, and the config, tokenizer and chat template that GGUF files carry in their metadata.

gpu

Accelerator backends: a trait the model’s linear layers dispatch through, plus a global registry chosen at start-up from the platform detection. Every backend is optional at run time; the CPU kernels remain the reference path and the fallback.

hf

Fetching and caching HuggingFace model repositories by name, over rsurl (pure-Rust HTTP).

janus

DeepSeek Janus-Pro (deepseek-ai/Janus-Pro-1B, -7B): one autoregressive model that both writes images and reads them.

kernels

CPU compute kernels: bf16 weights, f32 activations, multi-threaded via the spin pool.

lora

Runtime (unmerged) LoRA adapters for the language models: a PEFT adapter directory (adapter_config.json + adapter_model.safetensors) is attached to the linear layers it targets, and while its switch is on every targeted layer adds scale * B (A x) to its output. The base weights stay untouched, in whatever format they were loaded (bf16, Q8 / Q4, pre-quantized, MLX, GGUF), so one loaded model serves the base model (switch off: the code path is exactly the one without an adapter, bit for bit) and the adapted one (switch on).

media

Media decoding through oxideav (pure-Rust codecs): images to RGB8, audio to 16 kHz mono f32.

mm

Image, video and audio attachments: decoding (files through oxideav with the media feature, decoded oxideav frames with the oxideav feature), encoding with the vision / audio towers into soft tokens, and mapping them onto the placeholder tokens of the prompt. Gemma 4 takes images, videos (frames as images with mm:ss timestamps) and audio; Qwen3.5 / 3.6 / 3.8 take images and videos (temporal patches of two frames with <t seconds> timestamps, placed with M-RoPE).

mm_config

Multimodal (vision / audio) configuration from config.json.

model

Gemma 4 text decoder: embeddings, per-layer embeddings (PLE), sliding/global attention with KV sharing, gated-GELU MLP, tied softcapped LM head.

modernbert

ModernBERT encoders (modernbert-embed, gte-modernbert): pre-LayerNorm blocks without biases, a fused QKV projection with rotary positions, global attention every few layers and a bidirectional sliding window in the others (each with its own RoPE theta), a GeGLU MLP and a final LayerNorm. There are no position embeddings.

mpnet

MPNet encoders (all-mpnet-base-v2 and friends): a BERT-style post-LayerNorm encoder with RoBERTa-style absolute positions plus a T5-style relative position bias, one table of [buckets, heads] shared by every layer and added to the attention scores.

nomic_bert

NomicBERT encoders (model_type nomic_bert: nomic-embed-text v1 / v1.5, whose modeling code lives in nomic-ai/nomic-bert-2048): a BERT variant with rotary positions on a fused Wqkv, a gated MLP (fc2(fc11(x) * act(fc12(x)))), unbiased projections, LayerNorm after the embeddings and post-norm (or pre-norm) blocks, bidirectional attention.

platform

Runtime platform detection: CPU architecture and SIMD features, core count, memory, and GPUs (CUDA, Metal, Vulkan) probed by loading the vendor libraries at run time with dlsym, so the binary never links against them and runs everywhere with the CPU path as the fallback.

png

A dependency-free PNG writer for 8-bit RGB images (zlib “stored” blocks: no compression, so the file is about 3 bytes per pixel, but it works on every target, the browser included).

pool

Minimal spin-waiting thread pool for low-latency parallel-for.

prof

Lightweight always-on profiler: accumulated wall time per category.

quant

Int8 block quantization (Q8: blocks of 32 weights sharing one f32 scale).

qwen

Qwen3.5 / Qwen3.8 dense hybrid text model: Gated DeltaNet (linear attention with a causal short conv and a delta-rule recurrent state) interleaved with gated full attention, SiLU MLPs and zero-centered RMSNorm; plain Qwen3 (full attention only) runs on the same code, as do Llama 3 (full attention without query / key norms, llama3 RoPE scaling) and Qwen3.5 MoE (top-k routed SiLU experts plus a sigmoid-gated shared expert in place of the MLP). The vision encoder is in qwen_vision.rs; its tokens are placed with M-RoPE (interleaved time / height / width rotary positions, see QwenSession::add_vision_span). The MTP head for speculative decoding is in qwen_mtp.rs.

qwen_mtp

Multi-token prediction (“NextN”) speculative decoding for Qwen3.5 / Qwen3.8.

qwen_vision

Qwen3.5 / 3.6 / 3.8 vision (Qwen3_5ForConditionalGeneration’s model.visual): the image and video processors (Qwen2VLImageProcessor / Qwen3VLVideoProcessor: smart resize to multiples of 32 pixels within a pixel budget, uint8 bicubic, normalization to [-1, 1], 16x16 patches with 2 temporal frames each, emitted in 2x2 spatial-merge blocks), and the ViT: a Conv3d patch embedding, learned position embeddings bilinearly resampled to the patch grid, axial 2D RoPE, pre-LayerNorm blocks with biased projections and bidirectional attention within each frame, and the merger that folds every 2x2 block into one language-model token.

resample

Antialiased bicubic resizing of 8-bit RGB exactly as the Hugging Face processors run it: Pillow’s Image.resize(size, Image.BICUBIC) (the image processors’ PIL backend) and torchvision’s uint8 resize(..., BICUBIC, antialias=True) (the video processors, and the image processors’ torchvision backend).

safetensors

Minimal zero-copy safetensors reader (single file or sharded directory).

sampler

Token sampling: greedy, or temperature + top-k + top-p, optionally with a repetition penalty.

sd

Text-to-image with Stable Diffusion 1.x / 2.x and SD-Turbo from a diffusers folder (model_index.json, text_encoder/, tokenizer/, unet/, vae/, scheduler/).

snac

SNAC (Multi-Scale Neural Audio Codec, hubertsiuzdak/snac_24khz) decoder: codes of three codebooks at different rates (12, 23 and 47 Hz for the 24 kHz model) back to a waveform. This is what Orpheus TTS writes its audio tokens as (see crate::tts).

stream

Weight streaming for models larger than RAM: the weights are read in place from memory-mapped checkpoint files, and a reader thread pulls the byte ranges of the next layers into the page cache while the current one computes.

tools

Tool (function) calling: tool declarations in each model family’s own syntax, parsing of the calls a model emits, and a few built-in tools for the command line. The host runs the tools; chat::ChatFormat feeds the results back in the format the model was trained on.

torch

Reader for PyTorch torch.save checkpoints (pytorch_model.bin, *.pt): a zip archive whose entries are stored uncompressed, holding a pickled state dict (<prefix>/data.pkl) whose tensors point at raw storage files (<prefix>/data/<key>). Nothing is unpickled beyond what state dicts use, and no Python code is ever run: unknown callables are just opaque values.

tts

Text to speech with Orpheus TTS (canopylabs): a Llama 3.2 3B fine-tune that writes SNAC audio codes as extra vocabulary tokens, decoded to 24 kHz audio by the SNAC codec (crate::snac, hubertsiuzdak/snac_24khz by default).

video

Video input: frame sampling as the Hugging Face video processors do it, and (with the media feature) decoding through oxideav, from a video file or from a directory of frames. Decoded frames in memory are crate::av::VideoClip (the oxideav feature).

vision

Gemma 4 vision: image preprocessing (aspect-preserving bicubic resize into 16x16 patches), the ViT encoder with axial 2D RoPE and bidirectional attention, 3x3 patch pooling, and the projection of soft tokens into the language model’s embedding space.

whisper

Speech to text with OpenAI Whisper checkpoints in the Hugging Face layout (model_type: whisper): openai/whisper-{tiny,base,small,medium,large-v3,large-v3-turbo}, the .en models, and distil-whisper (same architecture, fewer decoder layers). Sizes come from config.json (80 or 128 mel bins), special tokens and decoding defaults from generation_config.json.