<div align="center">
<img src="https://raw.githubusercontent.com/Geekgineer/needle-rs/main/assets/banner.svg" alt="needle-rs" width="100%"/>
<br/><br/>
<p>
<a href="https://needle-rs.pages.dev"><b>→ Live demo</b></a>
</p>
<p>
<a href="https://github.com/Geekgineer/needle-rs/actions/workflows/ci.yml"><img src="https://github.com/Geekgineer/needle-rs/actions/workflows/ci.yml/badge.svg?branch=main" alt="CI"/></a>
<a href="https://github.com/Geekgineer/needle-rs/actions/workflows/release.yml"><img src="https://github.com/Geekgineer/needle-rs/actions/workflows/release.yml/badge.svg" alt="Release"/></a>
<a href="https://github.com/Geekgineer/needle-rs/actions/workflows/wasm-demo.yml"><img src="https://github.com/Geekgineer/needle-rs/actions/workflows/wasm-demo.yml/badge.svg?branch=main" alt="Demo"/></a>
<a href="#parity"><img src="https://img.shields.io/badge/parity-token--exact-brightgreen?style=flat-square" alt="Token-exact parity"/></a>
<a href="#quick-start"><img src="https://img.shields.io/badge/Needle-v1%20%2B%20v2-CE422B?style=flat-square" alt="Needle v1 and v2"/></a>
<a href="https://crates.io/crates/needle-infer"><img src="https://img.shields.io/crates/v/needle-infer?style=flat-square&color=CE422B" alt="crates.io"/></a>
<a href="https://www.npmjs.com/package/needle-rs"><img src="https://img.shields.io/npm/v/needle-rs?style=flat-square&color=CE422B" alt="npm"/></a>
<a href="https://pypi.org/project/needle-rs/"><img src="https://img.shields.io/pypi/v/needle-rs?style=flat-square&color=CE422B" alt="PyPI"/></a>
<a href="LICENSE"><img src="https://img.shields.io/badge/license-MIT-blue?style=flat-square" alt="MIT"/></a>
</p>
<p>
<a href="#quick-start">Quick start</a> ·
<a href="#how-it-works">How it works</a> ·
<a href="#parity">Parity</a> ·
<a href="https://github.com/cactus-compute/needle">Upstream model</a>
</p>
</div>
<br/>
<img src="https://raw.githubusercontent.com/Geekgineer/needle-rs/main/assets/terminal.svg" alt="A needle-rs session: two tool calls answered locally by Needle v2, then the same query on Needle v1" width="100%"/>
<p align="center">
<sub>Real output. For the browser version, try the <a href="https://needle-rs.pages.dev">live demo</a> — it runs both models.</sub>
</p>
<br/>
A pure-Rust + WebAssembly runtime for [Needle](https://github.com/cactus-compute/needle) by [Cactus Compute](https://github.com/cactus-compute) — small transformers that map `(query, tool list)` to a JSON function call. Deploys to browsers, edge workers, CLIs, Python, and `no_std` embedded targets. No server, no API key, no data leaving the device.
**Both model generations are supported in parallel:** Needle **v2** (45M, decoder-only, one 13.7 MB `.cact` file) and Needle **v1** (26M, encoder–decoder, SafeTensors + vocabulary). Same runtime, same API shape, one binary.
<br/>
<h2 id="why-this-matters">
<img src="https://img.shields.io/badge/-Why_this_matters-CE422B?style=flat-square" height="22" alt="Why this matters"/>
</h2>
Tool calling usually means a paid API round-trip or hundreds of megabytes on disk. This ships the whole agent in **14 MB** and runs it in a browser tab.
<table>
<thead>
<tr><th align="left">Stack</th><th align="right">Deploy size</th><th align="right">Cost</th><th align="center">Privacy</th><th align="center">Offline</th></tr>
</thead>
<tbody>
<tr><td>Hosted function calling</td><td align="right">SDK + API</td><td align="right">$ per token</td><td align="center">leaves device</td><td align="center">✗</td></tr>
<tr><td>llama.cpp + a 1B local model</td><td align="right">700 MB+</td><td align="right">free</td><td align="center">local</td><td align="center">✓</td></tr>
<tr><td>ONNX Runtime Web + a model</td><td align="right">8 MB + model</td><td align="right">free</td><td align="center">local</td><td align="center">✓</td></tr>
<tr><td><b><code>needle-rs</code> + Needle v2</b></td><td align="right"><b>414 KB + 13.7 MB</b></td><td align="right"><b>free</b></td><td align="center"><b>local</b></td><td align="center"><b>✓</b></td></tr>
</tbody>
</table>
The runtime is 414 KB of WebAssembly (163 KB over the wire, gzipped) with **one** runtime dependency. A generation session needs about 23 MB of working memory.
<br/>
<h2 id="quick-start">
<img src="https://img.shields.io/badge/-Quick_start-CE422B?style=flat-square" height="22" alt="Quick start"/>
</h2>
**Get a model.** v2 is a single self-describing file; v1 needs a vocabulary alongside its weights.
```bash
# Needle v2 — weights, geometry and tokenizer in one container
hf download Cactus-Compute/needle2 needle2.cact --local-dir weights/
# Needle v1
hf download Abdalrahman/needle-rs-safetensors needle.safetensors vocab.txt --local-dir weights/
```
<details open>
<summary><b>CLI</b> — <code>cargo install needle-rs-cli</code></summary>
<br/>
The crate is `needle-rs-cli`; the binary it installs is `needle-rs`. (Both
`needle-cli` and `needle-rs` on crates.io are unrelated projects — don't install
those.)
```bash
# v2 — the file extension selects the version, so there is no flag to get wrong
needle-rs --json --constrain weights/needle2.cact \
"What's the weather in Paris?" \
'[{"name":"get_weather","parameters":{"type":"object",
"properties":{"city":{"type":"string"}},"required":["city"]}}]'
# → [{"name":"get_weather","arguments":{"city":"Paris"}}]
# v1 — same binary, two files
needle-rs weights/needle.safetensors weights/vocab.txt "$QUERY" "$TOOLS"
```
</details>
<details open>
<summary><b>Rust</b> — <code>cargo add needle-infer</code></summary>
<br/>
```rust
use needle_infer::v2_engine::V2Engine; // Needle v2
let engine = V2Engine::load("weights/needle2.cact")?;
let out = engine.run(query, tools_json);
println!("{}", out.tool_call.unwrap_or(out.text));
use needle_infer::NeedleEngine; // Needle v1
let engine = NeedleEngine::load("weights/needle.safetensors", "weights/vocab.txt")?;
println!("{}", engine.run(query, tools_json).text);
```
</details>
<details open>
<summary><b>Browser / Node.js</b> — <code>npm install needle-rs</code></summary>
<br/>
```js
import init, { NeedleV2Wasm, NeedleWasm } from "needle-rs";
await init();
const v2 = NeedleV2Wasm.load(new Uint8Array(cactBytes));
const out = v2.run(query, toolsJson);
v2.run_json(query, toolsJson); // just the payload
v2.confidence_for(query, toolsJson, out); // confidence in that answer
v2.retrieve_tools(query, descriptions, 3); // rank tools by relevance
const v1 = NeedleWasm.load(weightsBytes, vocabText);
v1.run(query, toolsJson);
```
</details>
<details open>
<summary><b>Python</b> — <code>pip install needle-rs</code></summary>
<br/>
```python
from needle_rs import V2Engine, NeedleEngine
engine = V2Engine.load("weights/needle2.cact")
engine.run_json(query, tools_json) # tool-call payload
engine.generate(query, tools_json, constrain=True) # dict with stop_reason
engine.retrieve_tools(query, descriptions, top_k=3)
NeedleEngine.load("weights/needle.safetensors", "weights/vocab.txt")
```
One `abi3` wheel covers every CPython ≥ 3.8.
</details>
<br/>
<h2 id="versions">
<img src="https://img.shields.io/badge/-The_two_models-CE422B?style=flat-square" height="22" alt="The two models"/>
</h2>
Upstream replaced v1's encoder–decoder with a decoder-only architecture in a new container. They share no weights, loader, or quantisation scheme, so `needle-rs` implements both rather than migrating.
<table>
<thead><tr><th align="left"></th><th align="left">Needle v2</th><th align="left">Needle v1</th></tr></thead>
<tbody>
<tr><td>Parameters</td><td>45M</td><td>26M</td></tr>
<tr><td>Architecture</td><td>decoder-only; mHC lanes, Engram memory, HadamardMLP</td><td>encoder–decoder SAN</td></tr>
<tr><td>Weights</td><td>Cactus-Quants, ~2.2 bits effective</td><td>symmetric INT4</td></tr>
<tr><td>Files</td><td><b>one</b> <code>.cact</code> — 13.7 MB</td><td>22 MB + 122 KB vocabulary</td></tr>
<tr><td>Tokenizer</td><td>embedded in the container</td><td>separate file</td></tr>
<tr><td>Reasoning trace</td><td>✓ <code><think></code></td><td>—</td></tr>
<tr><td>Constrained decoding</td><td>optional</td><td>always on</td></tr>
<tr><td>Sampling</td><td>✓ temperature + seed</td><td>greedy only</td></tr>
<tr><td>Confidence head</td><td>✓</td><td>—</td></tr>
<tr><td>Tool retrieval head</td><td>✓ 128-d</td><td>✓</td></tr>
</tbody>
</table>
Every example in [`examples/`](examples/) runs on both.
<br/>
<h2 id="where-it-runs">
<img src="https://img.shields.io/badge/-Where_it_runs-CE422B?style=flat-square" height="22" alt="Where it runs"/>
</h2>
<table>
<thead><tr><th align="left">Target</th><th align="center">Status</th><th align="right">Binary</th></tr></thead>
<tbody>
<tr><td>Browser / Node.js / Cloudflare Workers <sub>(WASM)</sub></td><td align="center">✓</td><td align="right"><code>414 KB</code> <sub>163 KB gzipped</sub></td></tr>
<tr><td>Linux / macOS / Windows CLI</td><td align="center">✓</td><td align="right"><code>601 KB</code></td></tr>
<tr><td>Python <sub>(abi3 wheel, CPython ≥ 3.8)</sub></td><td align="center">✓</td><td align="right"><code>pip install needle-rs</code></td></tr>
<tr><td>C / C++ / Go / Swift <sub>(FFI)</sub></td><td align="center">✓</td><td align="right"><code>needle_v2_*</code> + <code>needle_*</code></td></tr>
<tr><td><code>no_std</code> embedded <sub>(Rust)</sub></td><td align="center">✓</td><td align="right"><sub>size varies</sub></td></tr>
<tr><td>iOS / Android, Apple & Snapdragon NPU</td><td align="center"><sub>use <a href="https://github.com/cactus-compute/cactus">Cactus</a></sub></td><td align="right"><sub>—</sub></td></tr>
</tbody>
</table>
Cactus's own engine targets mobile and NPUs with hand-tuned ARM SIMD. `needle-rs` targets everywhere else. MSRV is **1.87**.
<br/>
<h2 id="how-it-works">
<img src="https://img.shields.io/badge/-How_it_works-CE422B?style=flat-square" height="22" alt="How it works"/>
</h2>
<table>
<tr>
<td width="32" valign="top" align="center"><sub>1</sub></td>
<td valign="top"><b>Weights are never reconstructed.</b> A Cactus-Quants group dequantises as <code>w = u @ H</code> with <code>H</code> a normalised Walsh–Hadamard matrix. <code>H</code> is symmetric, so <code>dot(x, u @ H) == dot(H @ x, u)</code> — the rotation moves off the weights and onto the activation, paid once per matrix instead of once per row. At 512×512 that is 4 transforms instead of 512, and the inner loop becomes a dot product against packed bytes.</td>
</tr>
<tr>
<td valign="top" align="center"><sub>2</sub></td>
<td valign="top"><b>Fast Walsh–Hadamard, not a matmul.</b> Both places Needle uses <code>H</code> — quantisation groups and HadamardMLP — use a butterfly: <code>n log₂n</code> add/sub instead of <code>n²</code> multiply-accumulates.</td>
</tr>
<tr>
<td valign="top" align="center"><sub>3</sub></td>
<td valign="top"><b>The KV cache is a ring.</b> v2 attends over a 256-token window, so the cache holds 256 positions rather than <code>max_seq_len</code> — 14 MB instead of 113 MB.</td>
</tr>
<tr>
<td valign="top" align="center"><sub>4</sub></td>
<td valign="top"><b>Probe heads stream.</b> Confidence and retrieval pool over every layer's activations at every position — 117 MB if materialised. An online softmax reaches the same result in 16 KB.</td>
</tr>
<tr>
<td valign="top" align="center"><sub>5</sub></td>
<td valign="top"><b>Constrained decoding.</b> A character trie over declared tool names and argument keys, plus a JSON state machine, masks logits so the payload cannot name a tool that does not exist. Accepts both the flat and OpenAI schema styles.</td>
</tr>
</table>
Architecture deep-dive: [ARCHITECTURE.md](ARCHITECTURE.md) · v2 port record: [docs/v2-port-record.md](docs/v2-port-record.md).
<br/>
<h2 id="parity">
<img src="https://img.shields.io/badge/-Parity-7EE787?style=flat-square" height="22" alt="Parity"/>
</h2>
The failure mode for a from-scratch reimplementation is silent drift: output that looks right but diverges in the third decimal, producing rare and untraceable bugs. Both engines are held to the reference implementation's exact output.
**Needle v2** — verified against upstream's own `decode.forward_cached` running the same weights, reconstructed from the shipped container by [`tools/cact_params.py`](tools/cact_params.py):
| Forward pass, 788 captured intermediates across 27 layers | max relative deviation **1.9e-5** |
| End to end, 14 prompt/tool combinations (2,482 tokens) | **exact** token ids |
| Container: 145 CQ + 259 FP16 tensors, header, codebook | field-for-field match |
| Tokenizer, 44-case corpus | exact ids vs `RefTokenizer` **and** `sentencepiece` |
| Probe heads | contrastive **1.4e-6**, confidence **7.2e-5** |
| Batched + threaded prefill vs sequential | **bit-identical** |
**Needle v1** — 560 generated examples across five tool-name conventions, 0–8 parameters, 1–20 tools: **560/560 token-exact**.
Fixtures are committed, so the contract is version-pinned and reproducible without re-running Python. 253 Rust tests and 36 WASM binding assertions run in CI, in both the default and `parallel` feature configurations.
<br/>
<h2 id="benchmarks">
<img src="https://img.shields.io/badge/-Benchmarks-CE422B?style=flat-square" height="22" alt="Benchmarks"/>
</h2>
Needle v2 on an Apple M5 Max, steady state. Against the Python/JAX reference on the **same machine and the same weights**:
| Load model | **9 ms** | 761 ms + 26.5 s first-call JIT |
| Decode | **8.2 ms/token** | 11.0 ms/token |
| Prefill | 1.60 ms/token | **0.51 ms/token** |
| Session memory | **~23 MB** | — |
| Runtime dependencies | **0** | 4 |
Decode is **1.35× faster** and cold start about **50×** faster; prefill is slower, because the reference multiplies dense f32 weights while this runs from 2-bit packed ones. On a single query the two cross at **48 generated tokens** — faster above, slower below, and faster at any length on a cold process.
Prefill improved **3.6×** during the v2 port (5.74 → 1.60 ms/token) via batching, threading and batched Engram projections. The packed dot product is **2.9×** faster than a single-accumulator version and **9.1×** faster than a naive one.
Full methodology — including the optimisations that were measured and **rejected**, such as hand-written NEON losing to LLVM's autovectoriser — is in [BENCHMARKS.md](BENCHMARKS.md).
<br/>
<h2 id="use-cases">
<img src="https://img.shields.io/badge/-What_it's_good_for-CE422B?style=flat-square" height="22" alt="What it's good for"/>
</h2>
- **In-browser agents.** Route a user's sentence to one of your app's functions with no backend. See [`examples/browser-demo`](examples/browser-demo) and the [live demo](https://needle-rs.pages.dev).
- **Dynamic tool sets.** Generate tools from live state each turn and let the model pick — [`examples/dom-editor`](examples/dom-editor) rewrites a page from plain English.
- **Edge workers.** 414 KB of WASM fits inside a Cloudflare Worker.
- **Large tool catalogues.** Narrow hundreds of tools with the retrieval head before the call.
- **Uncertainty-aware routing.** Use the confidence head to escalate to a larger model only when needed.
- **Offline and embedded.** `no_std` kernels, one dependency, no allocator assumptions beyond `alloc`.
Not the right tool for open-ended chat, long-form generation, or reasoning beyond tool selection. It does one thing.
<br/>
<h2 id="acknowledgements">
<img src="https://img.shields.io/badge/-Acknowledgements-7D8590?style=flat-square" height="22" alt="Acknowledgements"/>
</h2>
Needle is designed and trained by [Henry Ndubuaku](https://github.com/hndubuaku) and the [Cactus Compute](https://github.com/cactus-compute) team. The model architecture, training code, dataset, and weights are entirely their work, released under MIT. `needle-rs` is an independent Rust runtime — no upstream code is copied, only the published architecture is implemented.
**If you find this useful, please star the [upstream Needle repo](https://github.com/cactus-compute/needle) as well.**
<br/>
<h2 id="citation">
<img src="https://img.shields.io/badge/-Citation-7D8590?style=flat-square" height="22" alt="Citation"/>
</h2>
The model is Cactus Compute's work. Cite it as they ask — this is their entry,
reproduced verbatim. The design and ablations are in the paper,
[arXiv:2607.18363](https://arxiv.org/abs/2607.18363).
```bibtex
@misc{needle2_2026,
title = {Needle 2: A 45M-Parameter Foundation Tool-Calling Model for Tiny Devices},
author = {Ndubuaku, Henry and Mosoyan, Karen and Mroz, Jakub and Cylich, Noah and
Kumar, Satyajit and Sandhu, Parkirat and Shemet, Roman and Lee, Justin H.},
year = {2026},
organization = {Cactus Compute, Inc.},
howpublished = {\url{https://github.com/cactus-compute/needle}}
}
```
If your work uses the v1 weights specifically, cite the v1 model instead:
```bibtex
@software{needle2026,
author = {Ndubuaku, Henry and {Cactus Compute}},
title = {Needle: A 26M-Parameter Tool-Calling Transformer},
year = {2026},
url = {https://github.com/cactus-compute/needle},
license = {MIT}
}
```
And this runtime, if it is relevant to what you are reporting:
```bibtex
@software{needlers2026,
author = {Ibrahim, Abdalrahman},
title = {needle-rs: Pure-Rust WASM Runtime for Needle},
year = {2026},
url = {https://github.com/geekgineer/needle-rs},
license = {MIT}
}
```
<br/>
<div align="center">
<sub>MIT — see <a href="LICENSE">LICENSE</a>. Model and weights by <a href="https://github.com/cactus-compute">Cactus Compute</a>, also MIT.</sub>
</div>