needle-core 0.3.0

no_std compute kernels for Needle: Cactus-Quants matmul, fast Walsh-Hadamard transform, attention, RoPE
Documentation

A pure-Rust + WebAssembly runtime for Needle by Cactus Compute — small transformers that map (query, tool list) to a JSON function call. Deploys to browsers, edge workers, CLIs, Python, and no_std embedded targets. No server, no API key, no data leaving the device.

All three model generations are supported in parallel: Needle 3 (121M, one 35.3 MB .cact file), Needle 2 (45M, 13.7 MB .cact) and Needle 1 (26M, SafeTensors + vocabulary). Same runtime, same API shape, one binary — the generation comes from the container, not the file name.

Tool calling usually means a paid API round-trip or hundreds of megabytes on disk. This ships the whole agent in 14 MB with Needle 2, or 36 MB with Needle 3, and runs it in a browser tab.

The runtime is 537 KB of WebAssembly (162 KB over the wire, brotli) with one runtime dependency, and carries all three model generations. A Needle 2 session needs about 23 MB of working memory; a Needle 3 session adds 8.8 MB of key/value cache at 512 tokens, or 2.3 MB with --kv-int8 — see choosing a generation.

All three run from the same binary, the same WASM module and the same Python package. The runtime picks the right engine from the container itself, so nothing depends on a file name.

Needle 3 Needle 2 Needle 1
Parameters 121M 45M 26M
Container needle3.cact, 35.3 MB needle2.cact, 13.7 MB .safetensors + vocab, 22.3 MB
Context 8192 2048 1024
KV cache, 512-token session 8.8 MB, or 2.3 MB at int8 3.5 MB —
Reasoning <think> on essentially every query <think> sometimes none
Confidence head ✓ ✓ ✗
Tool retrieval ✗ ✓ ✗ (not in the published weights)
Licence Apache-2.0 MIT MIT

Pick Needle 3 when quality matters most and you can afford a 35 MB download and roughly 9 MB of cache — or 2.3 MB with the int8 cache, which is the width the container declares and costs nothing in answer quality on our tests. It reasons before answering, which shows up on ambiguous queries and larger tool catalogues.

Pick Needle 2 for the smallest viable browser deployment, or when you need tool retrieval — v3 exports no contrastive head, so retrieve_tools and encode_contrastive are absent on the v3 API rather than present and always empty.

Needle 1 remains supported for compatibility. It abstains readily as a tool catalogue grows; prefer a newer generation for new work.

Needle 3's weights are Apache-2.0, where v1 and v2 are MIT. needle-rs itself is MIT in every case; the difference applies to the model.

Get a model. v2 is a single self-describing file; v1 needs a vocabulary alongside its weights.

# Needle v2 — weights, geometry and tokenizer in one container
hf download Cactus-Compute/needle2 needle2.cact --local-dir weights/

# Needle v1
hf download Abdalrahman/needle-rs-safetensors needle.safetensors vocab.txt --local-dir weights/

The crate is needle-rs-cli; the binary it installs is needle-rs. (Both needle-cli and needle-rs on crates.io are unrelated projects — don't install those.)

# v2 — the file extension selects the version, so there is no flag to get wrong
needle-rs --json --constrain weights/needle2.cact \
  "What's the weather in Paris?" \
  '[{"name":"get_weather","parameters":{"type":"object",
     "properties":{"city":{"type":"string"}},"required":["city"]}}]'
# → [{"name":"get_weather","arguments":{"city":"Paris"}}]

# v1 — same binary, two files
needle-rs weights/needle.safetensors weights/vocab.txt "$QUERY" "$TOOLS"

use needle_infer::v2_engine::V2Engine;          // Needle v2
let engine = V2Engine::load("weights/needle2.cact")?;
let out = engine.run(query, tools_json);
println!("{}", out.tool_call.unwrap_or(out.text));

use needle_infer::NeedleEngine;                 // Needle v1
let engine = NeedleEngine::load("weights/needle.safetensors", "weights/vocab.txt")?;
println!("{}", engine.run(query, tools_json).text);

import init, { NeedleV2Wasm, NeedleWasm } from "needle-rs";
await init();

const v2 = NeedleV2Wasm.load(new Uint8Array(cactBytes));
const out = v2.run(query, toolsJson);
v2.run_json(query, toolsJson);                 // just the payload
v2.confidence_for(query, toolsJson, out);      // confidence in that answer
v2.retrieve_tools(query, descriptions, 3);     // rank tools by relevance

const v1 = NeedleWasm.load(weightsBytes, vocabText);
v1.run(query, toolsJson);

from needle_rs import V2Engine, NeedleEngine

engine = V2Engine.load("weights/needle2.cact")
engine.run_json(query, tools_json)                     # tool-call payload
engine.generate(query, tools_json, constrain=True)     # dict with stop_reason
engine.retrieve_tools(query, descriptions, top_k=3)

NeedleEngine.load("weights/needle.safetensors", "weights/vocab.txt")

One abi3 wheel covers every CPython ≥ 3.8.

Upstream replaced v1's encoder–decoder with a decoder-only architecture in a new container, then rebuilt that again for v3 — hybrid local/global attention, a causal convolution over Q/K/V, five Engram sites and a reasoning step. No two generations share weights, loader or quantisation scheme, so needle-rs implements all three rather than migrating.

Every example in examples/ runs on all three.

Cactus's own engine targets mobile and NPUs with hand-tuned ARM SIMD. needle-rs targets everywhere else. MSRV is 1.87.

Architecture deep-dive: ARCHITECTURE.md · v2 port record: docs/v2-port-record.md.

The failure mode for a from-scratch reimplementation is silent drift: output that looks right but diverges in the third decimal, producing rare and untraceable bugs. All three engines are held to the reference implementation's exact output.

Needle v3 — verified against upstream's own model, with the tensor canon pinned by inverting export._tensors, since the container's directory is nameless and positional and per-tensor checks alone cannot catch a reordering:

What Result
Forward pass, 57 positions max relative deviation 9.0e-6, zero argmax mismatches
Incremental decode vs prefill bit-identical (0.000e0)
Container: 581 tensors, 196-byte header, codebook field-for-field match
Tokenizer, embedded SentencePiece exact ids vs sentencepiece
Engram hash indices exact as integers
Components: MLP 5.4e-6, attention 2.7e-6, confidence 2e-6 —
int8 KV cache vs upstream fake_quant exact, and prefill stays bit-identical to decode

Needle v2 — verified against upstream's own decode.forward_cached running the same weights, reconstructed from the shipped container by tools/cact_params.py:

What Result
Forward pass, 788 captured intermediates across 27 layers max relative deviation 1.9e-5
End to end, 14 prompt/tool combinations (2,482 tokens) exact token ids
Container: 145 CQ + 259 FP16 tensors, header, codebook field-for-field match
Tokenizer, 44-case corpus exact ids vs RefTokenizer and sentencepiece
Probe heads contrastive 1.4e-6, confidence 7.2e-5
Batched + threaded prefill vs sequential bit-identical

Needle v1 — 560 generated examples across five tool-name conventions, 0–8 parameters, 1–20 tools: 560/560 token-exact.

Fixtures are committed, so the contract is version-pinned and reproducible without re-running Python. 341 Rust tests and 94 WASM binding assertions run in CI, in both the default and parallel feature configurations, alongside the C ABI and Python wheel.

A measured caution: the reference config ships dtype="bfloat16", and a correct implementation scores 9.7 relative error against a bfloat16 oracle. Every figure above is against an f32 reference — see docs/v3-port-record.md.

Apple M5 Max, steady state, threaded — which is what the CLI, Python and C crates build. Needle v2 against the Python/JAX reference on the same machine and the same weights:

needle-rs Python / JAX
Load model 9 ms 761 ms + 26.5 s first-call JIT
Decode 8.2 ms/token 11.0 ms/token
Prefill 1.60 ms/token 0.51 ms/token
Session memory ~23 MB —
Runtime dependencies 0 4

Needle v3 carries 121M parameters against v2's 45M, so it costs roughly 2–3× per token — 10 ms to load, 4.20 ms/token prefill and 8.65 ms/token decode. A 100-token prompt answers in about 700 ms, 424 ms of it to first token. That is the trade v3 asks for, and the reason v2 is not deprecated.

Decode is 1.35× faster and cold start about 50× faster; prefill is slower, because the reference multiplies dense f32 weights while this runs from 2-bit packed ones. On a single query the two cross at 48 generated tokens — faster above, slower below, and faster at any length on a cold process.

Prefill improved 3.6× during the v2 port (5.74 → 1.60 ms/token) via batching, threading and batched Engram projections. The packed dot product is 2.9× faster than a single-accumulator version and 9.1× faster than a naive one.

Full methodology — including the optimisations that were measured and rejected, such as hand-written NEON losing to LLVM's autovectoriser — is in BENCHMARKS.md.

  • In-browser agents. Route a user's sentence to one of your app's functions with no backend. See examples/browser-demo and the live demo.
  • Dynamic tool sets. Generate tools from live state each turn and let the model pick — examples/dom-editor rewrites a page from plain English.
  • Edge workers. 537 KB of WASM fits inside a Cloudflare Worker.
  • Large tool catalogues. Narrow hundreds of tools with the retrieval head before the call.
  • Uncertainty-aware routing. Use the confidence head to escalate to a larger model only when needed.
  • Offline and embedded. no_std kernels, one dependency, no allocator assumptions beyond alloc.

Not the right tool for open-ended chat, long-form generation, or reasoning beyond tool selection. It does one thing.

Needle is designed and trained by Henry Ndubuaku and the Cactus Compute team. The model architecture, training code, dataset, and weights are entirely their work, released under MIT. needle-rs is an independent Rust runtime — no upstream code is copied, only the published architecture is implemented.

If you find this useful, please star the upstream Needle repo as well.

The model is Cactus Compute's work. Cite it as they ask — this is their entry, reproduced verbatim. The design and ablations are in the paper, arXiv:2607.18363.

@misc{needle2_2026,
  title        = {Needle 2: A 45M-Parameter Foundation Tool-Calling Model for Tiny Devices},
  author       = {Ndubuaku, Henry and Mosoyan, Karen and Mroz, Jakub and Cylich, Noah and
                  Kumar, Satyajit and Sandhu, Parkirat and Shemet, Roman and Lee, Justin H.},
  year         = {2026},
  organization = {Cactus Compute, Inc.},
  howpublished = {\url{https://github.com/cactus-compute/needle}}
}

If your work uses the v1 weights specifically, cite the v1 model instead:

@software{needle2026,
  author  = {Ndubuaku, Henry and {Cactus Compute}},
  title   = {Needle: A 26M-Parameter Tool-Calling Transformer},
  year    = {2026},
  url     = {https://github.com/cactus-compute/needle},
  license = {MIT}
}

And this runtime, if it is relevant to what you are reporting:

@software{needlers2026,
  author  = {Ibrahim, Abdalrahman},
  title   = {needle-rs: Pure-Rust WASM Runtime for Needle},
  year    = {2026},
  url     = {https://github.com/geekgineer/needle-rs},
  license = {MIT}
}