A pure-Rust + WebAssembly runtime for Needle by Cactus Compute — small transformers that map (query, tool list) to a JSON function call. Deploys to browsers, edge workers, CLIs, Python, and no_std embedded targets. No server, no API key, no data leaving the device.
All three model generations are supported in parallel: Needle 3 (121M, one 35.3 MB .cact file), Needle 2 (45M, 13.7 MB .cact) and Needle 1 (26M, SafeTensors + vocabulary). Same runtime, same API shape, one binary — the generation comes from the container, not the file name.
AI tool calling usually means a paid API round-trip or hundreds of megabytes on disk. This ships the whole agent in 14 MB with Needle 2, or 36 MB with Needle 3, and runs it in a browser tab.
The runtime is under 600 KB of WebAssembly (about 200 KB over the wire, brotli) with one runtime dependency, and carries all three model generations. A Needle 2 session needs about 23 MB of working memory; a Needle 3 session adds 8.8 MB of key/value cache at 512 tokens, or 2.3 MB with --kv-int8 — see choosing a generation.
All three run from the same binary, the same WASM module and the same Python package. The runtime picks the right engine from the container itself, so nothing depends on a file name.
| Needle 3 | Needle 2 | Needle 1 | |
|---|---|---|---|
| Parameters | 121M | 45M | 26M |
| Container | needle3.cact, 35.3 MB |
needle2.cact, 13.7 MB |
.safetensors + vocab, 22.3 MB |
| Context | 8192 | 2048 | 1024 |
| KV cache, 512-token session | 8.8 MB, or 2.3 MB at int8 | 3.5 MB | — |
| Reasoning | <think> on essentially every query |
<think> sometimes |
none |
| Confidence head | ✓ | ✓ | ✗ |
| Tool retrieval | ✗ | ✓ | ✗ (not in the published weights) |
| Licence | Apache-2.0 | MIT | MIT |
Pick Needle 3 when quality matters most and you can afford a 35 MB download and roughly 9 MB of cache — or 2.3 MB with the int8 cache, which is the width the container declares and costs nothing in answer quality on our tests. It reasons before answering, which shows up on ambiguous queries and larger tool catalogues.
Pick Needle 2 for the smallest viable browser deployment, or when you need
tool retrieval — the published Needle 3 weights export only a confidence head,
so retrieve_tools and encode_contrastive are absent on the v3 API rather
than present and always empty. Needle 3's architecture defines an embedding
head; it is not in this checkpoint, and would load unchanged if a future one
carried it.
Needle 1 remains supported for compatibility. It abstains readily as a tool catalogue grows; prefer a newer generation for new work.
Needle 3's weights are Apache-2.0, where v1 and v2 are MIT. needle-rs itself is MIT in every case; the difference applies to the model.
Get a model. v3 and v2 are each a single self-describing file; v1 needs a vocabulary alongside its weights.
# Needle v3 — weights, geometry and tokenizer in one container
# Needle v2 — smaller and faster, same one-file story
# Needle v1
The crate is needle-rs-cli; the binary it installs is needle-rs. (Both
needle-cli and needle-rs on crates.io are unrelated projects — don't install
those.)
# v3 — the container states its own generation, so there is no flag to get wrong
# → [{"name":"get_weather","arguments":{"city":"Paris"}}]
# Drop --json to see the <think> block v3 reasons with first.
# --kv-int8 stores the cache at 8 bits: 2.3 MB instead of 8.8 MB at 512 tokens.
# --layers runs a shallower rung of the same file: Needle 3 is trained so every
# depth from 2 to 20 blocks is a usable model. 8 blocks needs 3.5 MB of cache
# instead of 8.8 MB. Below 6 the answers stop being usable.
# v2 — same binary, same invocation
# v1 — same binary, two files
use V3Engine; // Needle v3
let engine = load?;
let text = engine.run; // reasoning + call
println!;
println!; // the <think> block, if any
// Pass the completion, never the bare query — the head scores a finished answer.
println!;
use V2Engine; // Needle v2
let engine = load?;
let out = engine.run;
println!;
use NeedleEngine; // Needle v1
let engine = load?;
println!;
import init from "needle-rs";
await ;
const v3 = ;
const out = v3.;
v3.; // just the payload
v3.; // the <think> block, or undefined
v3.; // pass the completion, not the query
v3.; // 8.8 MB — budget a tab before loading
v3.; // 2.3 MB at 8 bits
// No retrieve_tools on v3: these weights export only a confidence head.
const v2 = ;
v2.; // rank tools by relevance
const v1 = ;
v1.;
=
# tool-call payload
# dict: text, tool_call,
# reasoning, stop_reason
# 8-bit key/value cache
# the completion, not the query
# what a session will cost
# V3Engine has no retrieve_tools — these weights carry only a confidence head.
One abi3 wheel covers every CPython ≥ 3.8.
Upstream replaced v1's encoder–decoder with a decoder-only architecture in a new container, then rebuilt that again for v3 — hybrid local/global attention, a causal convolution over Q/K/V, five Engram sites and a reasoning step. No two generations share weights, loader or quantisation scheme, so needle-rs implements all three rather than migrating.
Every example in examples/ runs on all three.
Cactus's own engine targets mobile and NPUs with hand-tuned ARM SIMD. needle-rs targets everywhere else. MSRV is 1.87.
Architecture deep-dive: ARCHITECTURE.md · v2 port record: docs/v2-port-record.md.
The failure mode for a from-scratch reimplementation is silent drift: output that looks right but diverges in the third decimal, producing rare and untraceable bugs. All three engines are held to the reference implementation's exact output.
Needle v3 — verified against upstream's own model, with the tensor canon pinned by inverting export._tensors, since the container's directory is nameless and positional and per-tensor checks alone cannot catch a reordering:
| What | Result |
|---|---|
| Forward pass, 57 positions | max relative deviation 9.0e-6, zero argmax mismatches |
| Incremental decode vs prefill | bit-identical (0.000e0) |
| Container: 581 tensors, 196-byte header, codebook | field-for-field match |
| Tokenizer, embedded SentencePiece | exact ids vs sentencepiece |
| Engram hash indices | exact as integers |
| Components: MLP 5.4e-6, attention 2.7e-6, confidence 2e-6 | — |
int8 KV cache vs upstream fake_quant |
exact, and prefill stays bit-identical to decode |
| Ladder rungs, 2–20 blocks, against upstream's own slice | worst 1.258e-5 relative, zero argmax mismatches |
Needle v2 — verified against upstream's own decode.forward_cached running the same weights, reconstructed from the shipped container by tools/cact_params.py:
| What | Result |
|---|---|
| Forward pass, 788 captured intermediates across 27 layers | max relative deviation 1.9e-5 |
| End to end, 14 prompt/tool combinations (2,482 tokens) | exact token ids |
| Container: 145 CQ + 259 FP16 tensors, header, codebook | field-for-field match |
| Tokenizer, 44-case corpus | exact ids vs RefTokenizer and sentencepiece |
| Probe heads | contrastive 1.4e-6, confidence 7.2e-5 |
| Batched + threaded prefill vs sequential | bit-identical |
Needle v1 — 560 generated examples across five tool-name conventions, 0–8 parameters, 1–20 tools: 560/560 token-exact.
Fixtures are committed, so the contract is version-pinned and reproducible without re-running Python. 369 Rust tests and 100 WASM binding assertions run in CI, in both the default and parallel feature configurations, alongside the C ABI and Python wheel.
A measured caution, because it cost two debugging sessions: the reference config ships dtype="bfloat16", and measuring against a bfloat16 oracle makes a correct implementation look catastrophically wrong — a relative error of 9.7, i.e. about 970%, not a small number with a missing exponent. Every figure above is measured against an f32 reference. See docs/v3-port-record.md.
Apple M5 Max, steady state, threaded — which is what the CLI, Python and C crates build. Needle v2 against the Python/JAX reference on the same machine and the same weights:
| needle-rs | Python / JAX | |
|---|---|---|
| Load model | 9 ms | 761 ms + 26.5 s first-call JIT |
| Decode | 8.2 ms/token | 11.0 ms/token |
| Prefill | 1.60 ms/token | 0.51 ms/token |
| Session memory | ~23 MB | — |
| Runtime dependencies | 0 | 4 |
Decode is 1.35× faster and cold start about 50× faster; prefill is slower, because the reference multiplies dense f32 weights while this runs from 2-bit packed ones. On a single query the two cross at 48 generated tokens — faster above, slower below, and faster at any length on a cold process.
Prefill improved 3.6× during the v2 port (5.74 → 1.60 ms/token) via batching, threading and batched Engram projections. The packed dot product is 2.9× faster than a single-accumulator version and 9.1× faster than a naive one.
Needle v3, same machine and flags, carries 121M parameters against v2's 45M and costs roughly 2–3× per token: 10 ms to load, 4.20 ms/token prefill and 8.65 ms/token decode. A 100-token prompt answers in about 700 ms, 424 ms of it to first token. Its --kv-int8 cache costs about 2% of that speed and roughly a quarter of the memory. That is the trade v3 asks for, and the reason v2 is not deprecated.
Full methodology — including the optimisations that were measured and rejected, such as hand-written NEON losing to LLVM's autovectoriser — is in BENCHMARKS.md.
- In-browser agents. Route a user's sentence to one of your app's functions with no backend. See
examples/browser-demoand the live demo. - Dynamic tool sets. Generate tools from live state each turn and let the model pick —
examples/dom-editorrewrites a page from plain English. - Edge workers. The whole runtime fits inside a Cloudflare Worker.
- Large tool catalogues. Narrow hundreds of tools with the retrieval head before the call — Needle 2 only, which is the one generation that ships a contrastive head.
- Uncertainty-aware routing. Use the confidence head to escalate to a larger model only when needed.
- Offline and embedded.
no_stdkernels, one dependency, no allocator assumptions beyondalloc.
Not the right tool for open-ended chat, long-form generation, or reasoning beyond tool selection. It does one thing.
Needle is designed and trained by Henry Ndubuaku and the Cactus Compute team. The model architecture, training code, dataset, and weights are entirely their work, released openly — the upstream repository under Apache-2.0, the Needle 3 weights under Apache-2.0, and the Needle 2 and Needle 1 weights under MIT. needle-rs is an independent Rust runtime — no upstream code is copied, only the published architecture is implemented. See NOTICE.
If you find this useful, please star the upstream Needle repo as well.
The model is Cactus Compute's work. Cite it as they ask — these are their entries, reproduced verbatim from the upstream README. The design and ablations are in the paper, arXiv:2607.18363.
Current, and what to cite unless you mean an older generation specifically:
If your work uses the v2 weights specifically:
If your work uses the v1 weights specifically, cite the v1 model instead:
And this runtime, if it is relevant to what you are reporting: