TokenMiser

A local proxy that cuts your LLM bill. Point your OpenAI SDK at it and it routes easy work to free local models, serves repeats and near-repeats from cache, and collapses duplicate concurrent calls into one — while showing you exactly what you spent and saved.

Quick start

cargo install tokenmiser        # or: download a binary from Releases
tokenmiser serve                # proxy on :8443, dashboard at http://localhost:8443/

Change one line in your app:

from openai import OpenAI
client = OpenAI(base_url="http://localhost:8443/v1", api_key="unused")

That's the whole integration. With Ollama running you need no API keys at all — TokenMiser detects it and routes there for free. Add provider keys when you want frontier models.

Prebuilt binaries for macOS (Apple Silicon), Linux (x86_64 + aarch64) and Windows x64 are on the Releases page, each with a .sha256. Intel macs must build from source — the embedding runtime (ort) ships no prebuilt ONNX Runtime for x64 macOS.

What it actually saves

Measured on the repo's load battery (242 requests, 120-way concurrency, mixed streaming and non-streaming, against local Ollama):

Your savings depend entirely on your traffic: repetitive agent loops and RAG workloads with recurring questions benefit most; every-request-unique traffic benefits least. The dashboard tells you which you have rather than asking you to guess.

Why this exists

LiteLLM does routing. Portkey is closed source. Nothing open source combines routing, semantic caching, honest cost accounting and quality comparison in one binary you run yourself. TokenMiser is a Rust proxy that does — local-by-default, no signup, no telemetry, ever.

Using it

Pass model="auto" to let the router choose, or name any provider model directly. model="tokenmiser:cascade" tries a cheap model first and escalates only if the answer looks weak.

stream=True works as usual: chunks pass through as SSE, the finished response is cached, and a later cache hit replays as a stream rather than a single blob. client.models.list() returns the router pseudo-models plus every installed Ollama model.

Every response carries routing headers so you can audit each decision:

x-tokenmiser-cache: miss | l1-hit | l2-hit
x-tokenmiser-difficulty: easy | medium | hard
x-tokenmiser-tier: explicit | heuristic | semantic | cascade
x-tokenmiser-routed-to: <resolved model>
x-tokenmiser-budget: ok | exceeded          (when budget limits are configured)

Features

How it works

flowchart LR
    SDK[OpenAI SDK] -->|/v1/chat/completions| P[Pingora ingress :8443]
    P --> C1{L1 exact cache}
    C1 -->|hit| SDK
    C1 -->|miss| C2{L2 semantic cache}
    C2 -->|hit| SDK
    C2 -->|miss| R[Tiered router]
    R -->|easy| O[Ollama - free, local]
    R -->|medium| CHEAP[Cheap API model]
    R -->|hard| F[Frontier model]
    O & CHEAP & F --> L[Cost ledger] --> D[Dashboard /stats]

Configuration

Zero config required. To customize, point TOKENMISER_CONFIG at a YAML file:

listen:
  proxy_addr: "127.0.0.1:8443" # OpenAI-compatible ingress + dashboard
  admin_addr: "127.0.0.1:9443" # admin surface
                               # Loopback by default: this proxy is
                               # unauthenticated and spends your API budget.
                               # Bind 0.0.0.0 only if you really want anyone
                               # on your network to be able to use it.
providers:
  - name: ollama               # auto-detected if running; free tier
    kind: ollama
    base_url: "http://localhost:11434"
    default_model: llama3.2
  - name: openai               # used only if OPENAI_API_KEY is set
    kind: openai
    base_url: "https://api.openai.com/v1"
    api_key_env: OPENAI_API_KEY
  - name: anthropic            # used only if ANTHROPIC_API_KEY is set
    kind: anthropic
    base_url: "https://api.anthropic.com/v1"
    api_key_env: ANTHROPIC_API_KEY
routing:
  default_provider: ollama
cache:
  l1_enabled: true
  l2_enabled: true
  semantic_threshold: 0.87     # cosine similarity for L2 hits (precision-tuned)
  scope: tenant                # tenant | user | session | global
budget:                        # optional — omit for no budget tracking
  daily_usd: 5.0               # alert when today's spend (UTC) crosses this
  total_usd: 100.0             # alert when lifetime spend crosses this
  enforce: false               # true = reject paid-provider calls with 402 once exceeded
                               # (cache hits and local Ollama traffic always pass)
security:                      # optional — omit for the safe default
  allowed_origins: []          # browser origins allowed to drive the proxy.
                               # Empty = none (see "Browser safety" below).

Browser safety (CSRF)

TokenMiser listens on loopback with no authentication — the right default for a CLI/SDK tool, since only local processes can reach it. But your browser is a local process, and any page you visit can compose a request to it:

// on evil.example — no preflight, because text/plain is a CORS "simple request"
fetch("http://127.0.0.1:8443/v1/chat/completions", {
  method: "POST",
  headers: { "content-type": "text/plain" },
  body: JSON.stringify({ model: "gpt-5", messages: [{role:"user", content:"…"}] }),
});

CORS stops that page from reading the reply, but it never stopped the request — which already spent your API budget. So TokenMiser rejects requests carrying a cross-origin browser signal (Sec-Fetch-Site: cross-site, or an Origin that isn't allow-listed) with 403.

• curl, the OpenAI SDKs, and agent frameworks are unaffected — they send neither header, and are always allowed.

• The built-in dashboard keeps working — it is served by this same proxy, so its htmx polling is Sec-Fetch-Site: same-origin.

• A legitimate browser app opts in by listing its origin:

  security:
    allowed_origins: ["http://localhost:3000"]
  

  ["*"] disables the check entirely — this re-opens the hole for every page you visit, so prefer naming the origin.

Cost reporting honesty

Ollama Cloud models (any *-cloud tag, e.g. gpt-oss:20b-cloud) run on Ollama's paid servers, not your hardware — even though they are served through your local Ollama daemon. TokenMiser counts them as remote requests, and they are subject to budget.enforce like any other paid provider.

Their per-token price is not something TokenMiser can honestly encode, so rather than reporting $0.00 (which would read as "free"), those calls increment an explicit unpriced_requests counter in /stats and on the dashboard. When it is non-zero, spent_usd is a lower bound on your real bill, and the dashboard says so.

Verification

The test suite is fully mocked and never calls a paid API. Separately, the paid path has been exercised against the live OpenAI API (gpt-4o-mini), end-to-end through the proxy:

• Non-streaming and streaming (SSE) completions return correctly parsed, OpenAI-shaped responses, with unmodeled upstream fields passed through.
• Cost accounting matches the provider's own reported usage to the micro-dollar, checked against the published per-token price.
• Streamed responses are accounted at stream end and cached; the repeat is served from cache at $0.
• Budget enforcement was confirmed with real spend: warn mode surfaces x-tokenmiser-budget: exceeded, and enforce: true returns 402 on paid routes while local Ollama traffic and cache hits still pass.
• The router escalates to the paid provider only when the policy says so; a local-tier request adds no spend.
• An invalid or absent key produces a clean, OpenAI-shaped 401 with no key material in the response or logs.

Not yet verified against a live API: the Anthropic adapter, which is covered by unit tests only. Gemini and DeepSeek route through the OpenAI-compatible adapter but have not been exercised against their own endpoints.

Roadmap

• Pingora ingress + OpenAI-compatible /v1/chat/completions
• L1 exact cache + cost meter + canonical pricing.json
• L2 semantic cache (bge-small embeddings)
• Tier 0/1 router (heuristic + semantic classifier)
• Tier 2 speculative cascade
• Auto-detect Ollama + zero-config local routing
• Streaming SSE (pass-through, cache write on completion, cached hits replay as streams)
• Shadow-mode A/B + LLM-as-judge auto-gate
• Policy DSL (Rhai) + replay-test command
• MCP gateway + per-tool budgets
• Budget alerts (daily/total limits; warn by default, optional 402 enforcement)
• Single-flight dedup for concurrent identical misses (cache stampede)
• HNSW index for large L2 caches
• Multi-tenant persistence

Part of the Open Intelligence Labs ecosystem

• actants — local-first agent SDK; TokenMiser sits under it as the LLM layer
• DeepDive — research agent that routes search and analysis differently
• PhantomDep — blocks hallucinated dependencies before they install

Contributing

Issues and PRs welcome. Run cargo fmt, cargo clippy --workspace --all-targets, and cargo test --workspace before submitting.

License

MIT