llm-wasm: if your cost guard needs Tokio, it cannot run where the call starts
Why this exists: LLM calls now start in browsers, edge workers and plugins, and the logic that should guard them (what it costs, whether to retry, whether to block it) lives in SDKs that need an async runtime, a socket and a clock, so it gets skipped there.
Why this instead of the usual approach: SDK helpers are tied to their HTTP client. llm-wasm is the bookkeeping only, with no I/O and no clock: real prices and a cost ledger with budgets (llm-prices table), retry with jitter and Retry-After, request guards, model routing, a TTL cache, stream decoding and JSON repair. It builds for wasm32-unknown-unknown as is, with optional JavaScript bindings.
Why adopt it today: cargo add llm-wasm, wrap the call you already make with the ledger and the retry policy, and the same code enforces the budget in the browser and on the server.
Built for: developers calling LLMs from WebAssembly, Cloudflare/Fastly-style edge workers or plugin hosts, and anyone who wants one policy for both native and wasm.
Tradeoffs we chose on purpose:
- You bring the transport: it never sends a request.
- You pass the time in: no clock means TTLs and backoff take a timestamp argument.
The bookkeeping around an LLM API call (what it costs, whether to retry, whether to block it, which model to use, how to read a streamed answer, how to pull JSON out of it) as plain Rust that runs the same in a browser, an edge worker or a normal program.
In a browser or a Cloudflare Worker you usually cannot pull in Tokio or a full HTTP SDK. This crate does no I/O and reads no clock (you pass the time in), so it builds for wasm32-unknown-unknown unchanged. You make the HTTP call; llm-wasm decides whether to make it, which model to use, what it costs and how to read the answer. With the js feature it is also callable straight from JavaScript.
Install
# streaming, JSON repair, chat templates, secret redaction, JS bindings:
In ten lines
use CostLedger;
use JsonFormatter;
use ;
let req = new;
assert!; // OpenAI body
let reply = "Sure! {\"cities\": [\"Paris\", \"Lyon\"]} Anything else?";
let mut ledger = with_budget;
ledger.record.unwrap; // real price, $0.0000072
assert_eq!;
Quickstart (Rust)
use HashMap;
use ;
use ;
use JsonFormatter;
use ;
use RetryPolicy;
use ;
use TemplateEngine;
use ;
The same program is examples/quickstart.rs: cargo run --example quickstart.
Quickstart (JavaScript, feature js)
Build the module (needs wasm-bindgen-cli at the same version as the wasm-bindgen crate in your Cargo.lock, or use wasm-pack):
const llm = require;
llm.; // 0.00045 (USD)
llm.; // { a: [1, 2] }
llm.; // { a: [1, 2], ok: true } (feature repair)
// Stream an answer as it arrives (feature stream). Works for OpenAI, Anthropic
// and OpenAI-compatible APIs; pieces may split lines or characters anywhere.
const dec = ;
dec.; // { input_tokens, output_tokens } if sent
const ledger = ; // $1 budget
ledger.; // throws once over budget
const retry = ;
retry.;
const cache = ; // 60 s TTL, LRU, 1,000 entries
cache.;
llm.; // feature secrets
llm.; // feature jinja
Use --target web (and await init()) for browsers and workers.
Why this and not something else
- Provider SDKs (async-openai, the official JS SDKs) need an HTTP client and usually Tokio or Node; they do not build for a plain
wasm32-unknown-unknownworker. This crate is only the logic, so it runs wherever Rust compiles. - Doing it by hand is where the bugs are: this crate's 0.1 had a stack overflow on self-including templates, byte-counting length limits, a cache key two different prompts could share, and quadratic JSON extraction. 0.2 fixes those and is property-tested.
- Agent frameworks (rig, genai) are larger and do the HTTP for you; use them if you are on a server with Tokio.
Already using another crate or format?
- OpenAI-style chat APIs:
ChatRequestserializes to the Chat Completions body (rolein lowercase, unset options left out), and old capitalised JSON still reads. - OpenAI / Anthropic streaming:
StreamDecoderreads both SSE formats (parsing by sse-core). - Hugging Face models:
render_chat_templateruns thechat_templatefromtokenizer_config.jsonwith minijinja (Python string methods andraise_exceptionincluded), checked against the Llama 3 and Mistral templates. - LiteLLM: the price table is generated from LiteLLM's price list (
scripts/update_prices.py).
Features
| Feature | Default | What it adds | Extra dependencies | .wasm size with js (release, after wasm-bindgen, before wasm-opt) |
|---|---|---|---|---|
| (none) | yes | cost, retry, guards, routing, cache, templates, JSON extraction, types | serde, serde_json, thiserror, sha2, lru | |
js |
no | #[wasm_bindgen] exports: estimateCost, knownModels, extractJson, stripCodeFence, renderTemplate, checkRequest, routeModel, classes Ledger, Retry, Cache |
wasm-bindgen, serde-wasm-bindgen | 242 KB |
stream |
no | stream::StreamDecoder (+ JS StreamDecoder) |
sse-core, bytes | 273 KB |
repair |
no | JsonFormatter::extract_json_lenient (+ JS extractJsonLenient) |
json5 | 340 KB |
secrets |
no | secrets::SecretGuard: block or redact API keys, private keys, emails, card numbers (+ JS redactSecrets) |
regex | 672 KB |
jinja |
no | template::render_jinja, render_chat_template (+ JS renderChatTemplate) |
minijinja, minijinja-contrib | 1,578 KB |
| all of the above | 2,103 KB |
Examples
| Example | Shows | Run |
|---|---|---|
quickstart |
the whole request path: template, guards, routing, cache, retry, cost, JSON | cargo run --example quickstart |
stream_decode |
OpenAI and Anthropic streams cut into 7-byte pieces | cargo run --example stream_decode --features stream |
repair_json |
strict vs lenient extraction on five real-looking answers | cargo run --example repair_json --features repair |
chat_template |
Llama 3 prompt from its own template, with secret redaction first | cargo run --example chat_template --features jinja,secrets |
node/smoke.mjs |
every JavaScript export, from Node | see the JavaScript quickstart |
Modules
| Module | What it gives you |
|---|---|
cost |
pricing_for_model over 232 chat models (OpenAI, Anthropic, Gemini, Mistral, DeepSeek, xAI) from LiteLLM's price list (MIT); CostLedger with an optional hard USD budget and record_usd |
retry |
RetryPolicy: exponential backoff, delay_with_jitter (you pass a random number), delay_for_response (honours Retry-After seconds); retries 429 / 500 / 502 / 503 / 504 only |
guard |
Guard trait, ContentGuard (substring or whole_words, Unicode case-insensitive), LengthGuard (characters, not bytes), GuardChain that can block or rewrite a request |
secrets |
SecretGuard (feature secrets) |
routing |
Router with ordered RoutingRules (Always, MessageCountExceeds, ModelNameContains, MaxTokensBelow) and a fallback model |
cache |
TtlCache (time-to-live, least-recently-used eviction, caller supplies the time), cache_key(model, messages_json) (SHA-256) |
template |
TemplateEngine for {{variable}} and {{>partial}}; render_jinja / render_chat_template (feature jinja) |
format |
JsonFormatter::extract_json, extract_json_lenient (feature repair), MarkdownFormatter::strip_code_fence |
stream |
StreamDecoder (feature stream) |
types |
ChatMessage, ChatRequest, ChatResponse, StreamChunk, Role |
To refresh the price table: python scripts/update_prices.py, then rebuild. cost::PRICES_SNAPSHOT_DATE says when it was generated.
Performance
benches/vs_alternatives.rs (Intel i7-13700KF, Windows 11, release build, other work running, so rough):
| Task | llm-wasm 0.2 | comparison |
|---|---|---|
| extract JSON from a chatty 2 KB answer | 12.5 µs | 0.1.1: 13.4 µs |
extract JSON from text with 20,000 stray { |
3.9 ms (stops at a 4 MB scan budget) | 0.1.1: 187 ms (quadratic) |
| lenient extraction of an answer cut off halfway | 20.9 µs | |
render a 4-variable {{var}} template |
240 ns built-in | minijinja render_jinja: 2.5 µs (parses the template each call) |
| cache key + set + get with a 1 KB prompt | 563 ns (SHA-256, LRU) | 0.1.1: 969 ns (FNV-1a, HashMap) |
| decode a 500-chunk OpenAI stream in 64-byte pieces | 517 µs (86 MB/s) |
Reproduce with cargo bench --bench vs_alternatives --features repair,jinja,stream.
Design
- No
unwrap,expectorpanicin library code (denied by Clippy lints); every fallible call returnsLlmWasmError. Property tests feed random input to every parser. - Nothing reads the system clock or a random number generator, so behaviour is deterministic and identical on every target. Where randomness helps (retry jitter) you pass it in.
- Crate type is
cdylib+rlib, so the same crate is a Rust dependency and a.wasmmodule.
Limitations
extract_json_lenientdoes not insert missing commas between members, and gives up on JSON that only starts after more than 4 MB worth of scanning past stray brackets.{{var}}templates support substitution and partials only (nesting 32 deep); userender_jinjafor loops and conditionals.SecretGuardmatches known key formats; it is a safety net, not a guarantee.Retry-Afterin HTTP-date form is not parsed (there is no clock); the backoff is used instead.- Gemini models with long-context surcharges are priced at their base rate.
Run the tests with cargo test --all-features. Contributions are welcome, see CONTRIBUTING.md.
License
MIT, see LICENSE.