Skip to main content

Crate joshua

Crate joshua 

Source
Expand description

Joshua — a pure-Rust LLM inference engine.

Joshua provides a pure-Rust inference layer built on candle (HuggingFace’s native Rust ML framework) plus an OpenAI-compatible HTTP server built on axum. No C or C++ runtime dependencies are required for CPU inference.

§Quick start

use joshua::{Engine, GenerationOptions};

let engine = Engine::new("path/to/model.gguf").unwrap();

let messages = vec![joshua::ChatMessage::text("user", "Hello!")];

let (text, usage, _, _) = engine.complete(&messages, &GenerationOptions::default()).unwrap();
println!("{text}");

A tokenizer.json from the model’s HuggingFace repository must be placed alongside the .gguf file so the engine can tokenise prompts.

Re-exports§

pub use engine::find_gguf_in_dir;
pub use engine::Engine;
pub use engine::EngineOptions;
pub use engine::HugePages;
pub use engine::MlockMode;
pub use engine::MmapMode;
pub use engine::PageSize;
pub use error::JoshuaError;
pub use error::Result;
pub use types::ChatMessage;
pub use types::EmbeddingRequest;
pub use types::EmbeddingResponse;
pub use types::GenerationOptions;
pub use types::UsageInfo;

Modules§

compression
Detecting model files that cannot usefully be memory-mapped.
embedding
Dense text embeddings from GGUF models.
engine
Core LLM inference engine for Joshua.
error
Error types for the Joshua LLM inference engine.
gguf_ext
A GGUF header reader that tolerates quantization types candle does not know.
iq2xxs
IQ2_XXS — the 2.0625-bit trellis quantisation DeepSeek-V4-Flash GGUFs use for their routed expert gate/up projections.
kimi_k3
Kimi K3 (kimi-k3) — architecture primitives.
kquant_dot
Fused AVX2 dequant+dot kernels for candle’s k-quant block types used by the DeepSeek-V4-Flash GGUF (Q8_0, Q2_K, Q4_K).
mmap_tensor
Zero-copy quantized tensors borrowed directly from the memory-mapped GGUF.
model
Architecture-aware model dispatch for Joshua.
mxfp4
MXFP4 (OCP Microscaling FP4) support — GGML_TYPE_MXFP4, type id 39.
npu
NPU backend infrastructure.
quant_matmul
Threaded + AVX2-accelerated matmuls for candle’s k-quant block types (Q8_0, Q2_K, Q4_K, …).
quantized_deepseek2
Pure-Rust quantized loader for the deepseek2 GGUF architecture.
quantized_deepseek4
Pure-Rust quantized loader for the deepseek4 GGUF architecture.
server
OpenAI-compatible HTTP API server for Joshua.
simd
SIMD-accelerated f32 dot kernels (x86_64 AVX2/FMA) with runtime dispatch.
template
Chat-template rendering for GGUF models.
tools
Tool-call parsing for model output.
types
OpenAI-compatible request and response types used by Joshua.
whisper
Speech-to-text with Whisper — pure Rust.