Expand description
Joshua — a pure-Rust LLM inference engine.
Joshua provides a pure-Rust inference layer built on
candle (HuggingFace’s native Rust
ML framework) plus an OpenAI-compatible HTTP server built on axum.
No C or C++ runtime dependencies are required for CPU inference.
§Quick start
use joshua::{Engine, GenerationOptions};
let engine = Engine::new("path/to/model.gguf").unwrap();
let messages = vec![joshua::ChatMessage::text("user", "Hello!")];
let (text, usage, _, _) = engine.complete(&messages, &GenerationOptions::default()).unwrap();
println!("{text}");A tokenizer.json from the model’s HuggingFace repository must be placed
alongside the .gguf file so the engine can tokenise prompts.
Re-exports§
pub use engine::find_gguf_in_dir;pub use engine::Engine;pub use engine::EngineOptions;pub use engine::HugePages;pub use engine::MlockMode;pub use engine::MmapMode;pub use engine::PageSize;pub use error::JoshuaError;pub use error::Result;pub use types::ChatMessage;pub use types::EmbeddingRequest;pub use types::EmbeddingResponse;pub use types::GenerationOptions;pub use types::UsageInfo;
Modules§
- compression
- Detecting model files that cannot usefully be memory-mapped.
- embedding
- Dense text embeddings from GGUF models.
- engine
- Core LLM inference engine for Joshua.
- error
- Error types for the Joshua LLM inference engine.
- gguf_
ext - A GGUF header reader that tolerates quantization types candle does not know.
- iq2xxs
- IQ2_XXS — the 2.0625-bit trellis quantisation DeepSeek-V4-Flash GGUFs use for their routed expert gate/up projections.
- kimi_k3
- Kimi K3 (
kimi-k3) — architecture primitives. - kquant_
dot - Fused AVX2 dequant+dot kernels for candle’s k-quant block types used by the DeepSeek-V4-Flash GGUF (Q8_0, Q2_K, Q4_K).
- mmap_
tensor - Zero-copy quantized tensors borrowed directly from the memory-mapped GGUF.
- model
- Architecture-aware model dispatch for Joshua.
- mxfp4
- MXFP4 (OCP Microscaling FP4) support —
GGML_TYPE_MXFP4, type id 39. - npu
- NPU backend infrastructure.
- quant_
matmul - Threaded + AVX2-accelerated matmuls for candle’s k-quant block types (Q8_0, Q2_K, Q4_K, …).
- quantized_
deepseek2 - Pure-Rust quantized loader for the
deepseek2GGUF architecture. - quantized_
deepseek4 - Pure-Rust quantized loader for the
deepseek4GGUF architecture. - server
- OpenAI-compatible HTTP API server for Joshua.
- simd
- SIMD-accelerated f32 dot kernels (x86_64 AVX2/FMA) with runtime dispatch.
- template
- Chat-template rendering for GGUF models.
- tools
- Tool-call parsing for model output.
- types
- OpenAI-compatible request and response types used by Joshua.
- whisper
- Speech-to-text with Whisper — pure Rust.