needle-core 0.2.1

no_std compute kernels for Needle: Cactus-Quants matmul, fast Walsh-Hadamard transform, attention, RoPE
Documentation

A pure-Rust + WebAssembly runtime for Needle by Cactus Compute — small transformers that map (query, tool list) to a JSON function call. Deploys to browsers, edge workers, CLIs, Python, and no_std embedded targets. No server, no API key, no data leaving the device.

Both model generations are supported in parallel: Needle v2 (45M, decoder-only, one 13.7 MB .cact file) and Needle v1 (26M, encoder–decoder, SafeTensors + vocabulary). Same runtime, same API shape, one binary.

Tool calling usually means a paid API round-trip or hundreds of megabytes on disk. This ships the whole agent in 14 MB and runs it in a browser tab.

The runtime is 413 KB of WebAssembly (156 KB over the wire, brotli) with one runtime dependency. A generation session needs about 23 MB of working memory.

Get a model. v2 is a single self-describing file; v1 needs a vocabulary alongside its weights.

# Needle v2 — weights, geometry and tokenizer in one container
hf download Cactus-Compute/needle2 needle2.cact --local-dir weights/

# Needle v1
hf download Abdalrahman/needle-rs-safetensors needle.safetensors vocab.txt --local-dir weights/

The crate is needle-rs-cli; the binary it installs is needle-rs. (Both needle-cli and needle-rs on crates.io are unrelated projects — don't install those.)

# v2 — the file extension selects the version, so there is no flag to get wrong
needle-rs --json --constrain weights/needle2.cact \
  "What's the weather in Paris?" \
  '[{"name":"get_weather","parameters":{"type":"object",
     "properties":{"city":{"type":"string"}},"required":["city"]}}]'
# → [{"name":"get_weather","arguments":{"city":"Paris"}}]

# v1 — same binary, two files
needle-rs weights/needle.safetensors weights/vocab.txt "$QUERY" "$TOOLS"

use needle_infer::v2_engine::V2Engine;          // Needle v2
let engine = V2Engine::load("weights/needle2.cact")?;
let out = engine.run(query, tools_json);
println!("{}", out.tool_call.unwrap_or(out.text));

use needle_infer::NeedleEngine;                 // Needle v1
let engine = NeedleEngine::load("weights/needle.safetensors", "weights/vocab.txt")?;
println!("{}", engine.run(query, tools_json).text);

import init, { NeedleV2Wasm, NeedleWasm } from "needle-rs";
await init();

const v2 = NeedleV2Wasm.load(new Uint8Array(cactBytes));
const out = v2.run(query, toolsJson);
v2.run_json(query, toolsJson);                 // just the payload
v2.confidence_for(query, toolsJson, out);      // confidence in that answer
v2.retrieve_tools(query, descriptions, 3);     // rank tools by relevance

const v1 = NeedleWasm.load(weightsBytes, vocabText);
v1.run(query, toolsJson);

from needle_rs import V2Engine, NeedleEngine

engine = V2Engine.load("weights/needle2.cact")
engine.run_json(query, tools_json)                     # tool-call payload
engine.generate(query, tools_json, constrain=True)     # dict with stop_reason
engine.retrieve_tools(query, descriptions, top_k=3)

NeedleEngine.load("weights/needle.safetensors", "weights/vocab.txt")

One abi3 wheel covers every CPython ≥ 3.8.

Upstream replaced v1's encoder–decoder with a decoder-only architecture in a new container. They share no weights, loader, or quantisation scheme, so needle-rs implements both rather than migrating.

Every example in examples/ runs on both.

Cactus's own engine targets mobile and NPUs with hand-tuned ARM SIMD. needle-rs targets everywhere else. MSRV is 1.87.

Architecture deep-dive: ARCHITECTURE.md · v2 port record: docs/v2-port-record.md.

The failure mode for a from-scratch reimplementation is silent drift: output that looks right but diverges in the third decimal, producing rare and untraceable bugs. Both engines are held to the reference implementation's exact output.

Needle v2 — verified against upstream's own decode.forward_cached running the same weights, reconstructed from the shipped container by tools/cact_params.py:

What Result
Forward pass, 788 captured intermediates across 27 layers max relative deviation 1.9e-5
End to end, 14 prompt/tool combinations (2,482 tokens) exact token ids
Container: 145 CQ + 259 FP16 tensors, header, codebook field-for-field match
Tokenizer, 44-case corpus exact ids vs RefTokenizer and sentencepiece
Probe heads contrastive 1.4e-6, confidence 7.2e-5
Batched + threaded prefill vs sequential bit-identical

Needle v1 — 560 generated examples across five tool-name conventions, 0–8 parameters, 1–20 tools: 560/560 token-exact.

Fixtures are committed, so the contract is version-pinned and reproducible without re-running Python. 253 Rust tests and 36 WASM binding assertions run in CI, in both the default and parallel feature configurations.

Needle v2 on an Apple M5 Max, steady state. Against the Python/JAX reference on the same machine and the same weights:

needle-rs Python / JAX
Load model 9 ms 761 ms + 26.5 s first-call JIT
Decode 8.2 ms/token 11.0 ms/token
Prefill 1.60 ms/token 0.51 ms/token
Session memory ~23 MB —
Runtime dependencies 0 4

Decode is 1.35× faster and cold start about 50× faster; prefill is slower, because the reference multiplies dense f32 weights while this runs from 2-bit packed ones. On a single query the two cross at 48 generated tokens — faster above, slower below, and faster at any length on a cold process.

Prefill improved 3.6× during the v2 port (5.74 → 1.60 ms/token) via batching, threading and batched Engram projections. The packed dot product is 2.9× faster than a single-accumulator version and 9.1× faster than a naive one.

Full methodology — including the optimisations that were measured and rejected, such as hand-written NEON losing to LLVM's autovectoriser — is in BENCHMARKS.md.

  • In-browser agents. Route a user's sentence to one of your app's functions with no backend. See examples/browser-demo and the live demo.
  • Dynamic tool sets. Generate tools from live state each turn and let the model pick — examples/dom-editor rewrites a page from plain English.
  • Edge workers. 413 KB of WASM fits inside a Cloudflare Worker.
  • Large tool catalogues. Narrow hundreds of tools with the retrieval head before the call.
  • Uncertainty-aware routing. Use the confidence head to escalate to a larger model only when needed.
  • Offline and embedded. no_std kernels, one dependency, no allocator assumptions beyond alloc.

Not the right tool for open-ended chat, long-form generation, or reasoning beyond tool selection. It does one thing.

Needle is designed and trained by Henry Ndubuaku and the Cactus Compute team. The model architecture, training code, dataset, and weights are entirely their work, released under MIT. needle-rs is an independent Rust runtime — no upstream code is copied, only the published architecture is implemented.

If you find this useful, please star the upstream Needle repo as well.

The model is Cactus Compute's work. Cite it as they ask — this is their entry, reproduced verbatim. The design and ablations are in the paper, arXiv:2607.18363.

@misc{needle2_2026,
  title        = {Needle 2: A 45M-Parameter Foundation Tool-Calling Model for Tiny Devices},
  author       = {Ndubuaku, Henry and Mosoyan, Karen and Mroz, Jakub and Cylich, Noah and
                  Kumar, Satyajit and Sandhu, Parkirat and Shemet, Roman and Lee, Justin H.},
  year         = {2026},
  organization = {Cactus Compute, Inc.},
  howpublished = {\url{https://github.com/cactus-compute/needle}}
}

If your work uses the v1 weights specifically, cite the v1 model instead:

@software{needle2026,
  author  = {Ndubuaku, Henry and {Cactus Compute}},
  title   = {Needle: A 26M-Parameter Tool-Calling Transformer},
  year    = {2026},
  url     = {https://github.com/cactus-compute/needle},
  license = {MIT}
}

And this runtime, if it is relevant to what you are reporting:

@software{needlers2026,
  author  = {Ibrahim, Abdalrahman},
  title   = {needle-rs: Pure-Rust WASM Runtime for Needle},
  year    = {2026},
  url     = {https://github.com/geekgineer/needle-rs},
  license = {MIT}
}