Please check the build logs for more information.
See Builds for ideas on how to fix a failed build, or Metadata for how to configure docs.rs builds.
If you believe this is docs.rs' fault, open an issue.
馃敺 Prism Inference
A Rust CPU runtime for Prism-compatible language models.
GGUF models 路 Packed quantization 路 Local inference
Download 路 Documentation 路 Roadmap
Prism Inference runs language models on the CPU using Rust, with packed quantized weights, GGUF loading, tokenization and streamed output. It links no C/C++ inference library.
The first generation path supports the dense Qwen35 architecture used by Swift/Bonsai. Broader Prism architectures and general model selection are in scope and planned. Prism Inference is an independent implementation, separate from PrismML's upstream runtime.
[!IMPORTANT] Early development. Bounded original-PQ2 Swift workloads have retained numerical evidence. Broader full-model, PTQ1/adapter and device qualification remain open. See the current status and roadmap before choosing it for an application.
At a glance
| Current support | |
|---|---|
| Runtime | Rust CPU inference with bounded model and conversation storage |
| First model path | Swift/Bonsai 路 dense Qwen35 路 Prism GGUF |
| Acceleration | Scalar reference kernels; eligible Windows ARM64 NEON |
| Windows bundles | ARM64 and x64 executables, licenses and integrity receipts |
Get started
Download a release bundle for your Windows architecture, extract it, and start with:
.\prism-inference.exe --help
Bundles contain prebuilt executables, so running them needs no Rust build. Models are separate downloads; the CLI guide explains model identity, input options and resource budgets. SwiftBonsai provides model setup and a guarded launcher that consumes the prebuilt private server.
For source installation or Rust library integration, follow the
build and library guide.
It pins the reviewed Git source and compiler. GitHub binary releases and
crates.io versions are separate deliveries; an unpinned cargo install can
select an older runtime.
Included tools
| Executable | Purpose |
|---|---|
prism-inference |
Generate from text or token IDs; inspect tokenization |
bonsai-token-worker |
Persistent token counting and encoding over JSONL |
bonsai-swift-worker |
Swift model inference over JSONL |
prism-inference-server |
Private HTTP inference for an owning launcher |
bonsai-gguf |
GGUF inspection, PQ2 census and PTQ1 export |
Documentation
| Guide | What you'll find |
|---|---|
| Documentation index | Where to start for each task |
| CLI and workers | Inputs, streaming, tokenization and service protocols |
| Runtime contracts | Implemented behavior, memory, ownership and SIMD |
| Building and validation | Source builds, tests, CI and release bundles |
| Roadmap | Current evidence, broader model support and remaining work |
License
Original code and documentation use the Apache License 2.0. Third-party material retains its terms, including Prism/ggml and Camelid MIT attributions in NOTICE. Model weights and adapters are separate artifacts governed by their publishers' licenses.