<div align="center">
<img src=".github/prints/logo.jpg" alt="tokenix logo" width="560" style="max-width: 100%; border-radius: 14px; box-shadow: 0 8px 32px rgba(0, 0, 0, 0.2);" />
<h1>tokenix</h1>
<p><strong>Give your AI coding agent the exact slice of the repo it needs — not the whole file.</strong></p>
<p><em>Local code index · Tree-sitter symbol graph · Surgical reads · Deterministic output filters · Native agent hooks</em></p>
<p>
<a href="https://github.com/juninmd/tokenix/releases"><img src="https://img.shields.io/github/v/release/juninmd/tokenix?style=flat-square&color=ff7700&label=release" alt="Latest Release" /></a>
<a href="https://crates.io/crates/tokenix"><img src="https://img.shields.io/crates/v/tokenix?style=flat-square&color=e05d44&label=crates.io" alt="crates.io" /></a>
<a href="https://crates.io/crates/tokenix"><img src="https://img.shields.io/crates/d/tokenix?style=flat-square&color=blue&label=downloads" alt="crates.io downloads" /></a>
<a href="https://github.com/juninmd/tokenix/stargazers"><img src="https://img.shields.io/github/stars/juninmd/tokenix?style=flat-square&color=yellow" alt="GitHub stars" /></a>
<a href="https://github.com/juninmd/tokenix/actions/workflows/rust.yml"><img src="https://img.shields.io/github/actions/workflow/status/juninmd/tokenix/rust.yml?branch=main&style=flat-square&label=CI" alt="CI" /></a>
<a href="https://scorecard.dev/viewer/?uri=github.com/juninmd/tokenix"><img src="https://img.shields.io/ossf-scorecard/github.com/juninmd/tokenix?style=flat-square&label=scorecard" alt="OpenSSF Scorecard" /></a>
<a href="https://github.com/juninmd/tokenix/blob/main/LICENSE"><img src="https://img.shields.io/badge/license-MIT-blue?style=flat-square" alt="License" /></a>
<a href="https://www.rust-lang.org/"><img src="https://img.shields.io/badge/built%20with-Rust%20(MSRV%201.90)-dea584?style=flat-square&logo=rust" alt="Built with Rust" /></a>
<img src="https://img.shields.io/badge/platform-Linux%20%7C%20macOS%20%7C%20Windows-555555?style=flat-square" alt="Platforms" />
</p>
<p>
<a href="#-quick-start"><b>⚡ Quick start</b></a> ·
<a href="#-features-at-a-glance"><b>✨ Highlights</b></a> ·
<a href="#-using-tokenix-day-to-day"><b>🧭 Guide</b></a> ·
<a href="#-agent-guides"><b>🔌 Agent guides</b></a> ·
<a href="#-dashboard"><b>🖥️ Dashboard</b></a> ·
<a href="#-command-reference"><b>📖 Commands</b></a> ·
<a href="#-troubleshooting"><b>❓ Troubleshooting</b></a>
</p>
</div>
---
> ⚡ **tokenix** is a high-performance Rust CLI that sits directly between your AI coding agent and your repository. It indexes your codebase locally, and it never changes what the agent reads. It trims verbose command output (a 10,000-line test run keeps its errors) and offers a symbol graph and outlines the agent can call when it wants them.
```text
┌─────────────────────────────────────────────────────────────────────────────────────────────┐
│ Native Read(src/hook.rs) → 1,518 lines → 13,498 tokens (always allowed, untouched) │
│ tokenix read src/hook.rs → outline → 2,395 tokens (opt-in; omits usages/tests) │
└─────────────────────────────────────────────────────────────────────────────────────────────┘
```
## ✨ Features at a glance
| A SQLite index of every file, chunked by symbol. No model to download, no background process, zero external services. | Deep AST symbol parsing with PageRank importance, bidirectional caller/callee tracing, and diff blast-radius impact analysis. | 528+ bundled filters stripping noisy build/test logs without masking real errors or exit codes. |
| In-flight credential masking on hooks and failure tees. Repo-controlled filters are isolated behind SHA-256 trust gates. | 40% to 90% real payload reduction before prompt-cache writes. 100% fail-open contract (always exit 0 on errors). | Native hook support for Claude Code, OpenAI Codex, GitHub Copilot, Antigravity, OpenCode, and any MCP client. |
---
## ⚡ Quick start
> [!NOTE]
> Four simple steps, about three minutes. Nothing is downloaded besides the binary.
### 1️⃣ Install the binary
```bash
# macOS / Linux: pick the asset for your platform (table below)
curl -fsSL https://github.com/juninmd/tokenix/releases/latest/download/tokenix-linux-x86_64 -o tokenix
chmod +x tokenix && sudo mv tokenix /usr/local/bin/tokenix
```
```powershell
# Windows
irm https://github.com/juninmd/tokenix/releases/latest/download/tokenix-windows-x86_64.exe -OutFile tokenix.exe
.\tokenix.exe install-binary # copies itself to a per-user bin dir on PATH
```
### 2️⃣ Index your repository
```bash
cd your-project
tokenix index .
```
### 3️⃣ Connect your agent
```bash
tokenix install-hook --tool claude-code # or codex · copilot · antigravity · opencode · all
```
Each agent is wired a little differently. The [agent guides](#-agent-guides) cover
what gets installed, what gets intercepted and how to undo it.
### 4️⃣ Verify that it works
```bash
tokenix doctor # binary, filters, recordings
tokenix # dashboard → Stats tab: which agents are wired
tokenix gain # after a session: hook calls, intercepts, tokens removed
```
Then work with your agent as usual. You do not need to change your prompts or habits.
<details>
<summary><b>All install options</b> (platform assets, crates.io, source, verification)</summary>
Every release ships one binary per platform. The version-less
`releases/latest/download/<asset>` URL **always resolves to the newest release**.
| Linux x86_64 | `tokenix-linux-x86_64` |
| Linux arm64 (aarch64) | `tokenix-linux-aarch64` |
| macOS (Apple Silicon / aarch64) | `tokenix-macos-aarch64` |
| Windows x86_64 | `tokenix-windows-x86_64.exe` |
*(Intel macOS `x86_64` users can install from source using `cargo install tokenix --locked`)*
Each release also publishes `sha256sums.txt` and SLSA build-provenance attestations.
[SECURITY.md](SECURITY.md) explains how to verify a download.
```bash
cargo install tokenix --locked # from crates.io
git clone https://github.com/juninmd/tokenix && cd tokenix
cargo install --path . --locked # from source
```
Use `--locked`. It builds against the committed `Cargo.lock`. Without it, `cargo
install` re-resolves dependencies and can pull an incompatible `ureq` into the
`ort-sys` build script. Building from source needs a recent stable
[Rust](https://www.rust-lang.org/tools/install) toolchain (MSRV 1.90).
> [!TIP]
> **For AI agents installing tokenix:** Use the prebuilt binary, not `cargo install`. It needs no toolchain and no compile step. Fetch the version-less URL and **never hard-code a version**. Detect the platform, download the matching asset, mark it executable, then run `tokenix doctor`.
</details>
---
## 🧭 Using tokenix day to day
### 1. Let the agent use it automatically
After `install-hook`, tokenix sees each tool call before it runs and decides:
| Read any file | Nothing: `Read` is never intercepted | The agent gets the file. A measured A/B showed outlines hiding usages during renames |
| Grep, any pattern | Nothing (only the `head_limit` cap below) | The agent's own grep runs as asked |
| Grep with unbounded content output | Adds `head_limit` (100 by default) | Output is capped |
| Run a noisy command (`cargo test`, `terraform plan`, `git status` …) | Reruns it through `tokenix run` and the matching filter | Failures and summaries stay, noise goes, and the **exit code is the real one** |
Anything else passes through untouched. Nothing is lost for good: see
[getting the full output back](#5-when-you-need-the-raw-output).
### 2. Ask it questions yourself
The same commands the agent uses work in your terminal. Here they run against a
two-file demo project:
```console
$ tokenix symbols apply_tax
## Symbols matching `apply_tax`
- src/billing.rs:5-7 [function] apply_tax
$ tokenix callers apply_tax
## Callers of `apply_tax`
- src/billing.rs:1 [function] invoice_total -> src/billing.rs:5 [function] apply_tax via `apply_tax` (references)
$ tokenix read src/billing.rs --symbol apply_tax
# L5-7 [function] apply_tax
fn apply_tax(amount: u32) -> u32 {
amount + amount / 10
}
```
| Find a symbol by name | `tokenix symbols apply_tax` |
| See a big file's structure | `tokenix read src/billing.rs` |
| Read exactly one function | `tokenix read src/billing.rs --symbol apply_tax` |
| Find who calls something, and what it calls | `tokenix callers apply_tax` · `tokenix callees main` |
| See everything a change could break | `tokenix impact apply_tax` · `tokenix flow main` |
| Hand a budgeted bundle to a tool without hooks | `tokenix pack --budget 8000 > context.md` |
### 3. Review a change before you push
`blast` maps your diff to the symbols it touches and walks the call graph up from
them:
```console
$ tokenix blast # vs HEAD; use --since origin/main for a branch
# Blast radius vs HEAD — 1 changed file(s), 1 changed symbol(s)
## Changed symbols
- apply_tax (function) src/billing.rs:5
## Impacted (up to 2 hop(s) of callers)
- [1] invoice_total (↑1) src/billing.rs:1
- [2] main (↑0) src/main.rs:11
```
### 4. Keep the index current
- **Edits are picked up on their own.** Before answering, every index command
re-chunks the files you changed since the last index. That takes milliseconds
so a function you just wrote is searchable right away. Run `tokenix index`
now and then (or `tokenix index --if-stale` in a script) for larger changes.
- **Branch switches are detected.** The index records the branch and `HEAD`. When
they change, the hook passes calls through until you re-index, so it never
answers from the wrong branch.
- **Share one index with the team.** `tokenix export-index` writes
`.tokenix/index.db.gz` with known secrets masked. Teammates run
`tokenix import-index` instead of indexing from zero.
### 5. When you need the raw output
- A clipped successful run ends with `[tokenix: N bytes not shown — tokenix retrieve <key> …]`.
`tokenix retrieve <key>` prints the exact original bytes.
- A failed run tees its full output, with credentials masked, to `~/.tokenix/tee/`,
and prints the path.
- `tokenix run --raw "cmd"` (or `TOKENIX_RAW=1`) passes output through byte for byte.
Use it when a script parses the output.
- `TOKENIX_DISABLED=1 <cmd>` bypasses the hook for one command.
### 6. Tune it for your project
Put a `.tokenix.toml` in the repository root. Unknown keys are reported, not
silently ignored.
```toml
[index]
exclude = ["fixtures", "vendor"] # extra directories to skip
extensions = ["proto"] # extra extensions to index
data_files = false # index .json/.yaml/.yml too
redact_secrets = true # mask secret-shaped strings in stored chunks
max_file_bytes = 1500000 # files above this are skipped whole
```
### 7. Add your own output filters
A filter is a TOML rule, never a model. Start from your real output:
```bash
tokenix discover # which commands in your history cost the most
tokenix filter record "npm test" # capture real output (redacted, git-ignored)
tokenix filter generate npm # draft a filter with an AI CLI you have installed
tokenix filter verify # run its golden tests through the real pipeline
```
User filters go in `~/.tokenix/filters/`. Filters committed to a repo in
`.tokenix/filters/` apply **only after `tokenix trust`**, and editing one revokes
trust until you approve it again. [Output filters](#-output-filters) covers the
format.
---
## 🔌 Agent guides
| **Claude Code** | `tokenix install-hook --tool claude-code` | `PreToolUse` hook in `settings.json` | [docs/agents/claude-code.md](docs/agents/claude-code.md) |
| **OpenAI Codex CLI** | `tokenix install-hook --tool codex` | `PreToolUse` hook for commands + `tx-read`/`tx-symbols` helpers | [docs/agents/codex.md](docs/agents/codex.md) |
| **GitHub Copilot** | `tokenix install-hook --tool copilot` | `PreToolUse` + `PostToolUse` hooks; `--local` commits them to `.github/` | [docs/agents/copilot.md](docs/agents/copilot.md) |
| **Antigravity** | `tokenix install-hook --tool antigravity` | Native plugin | [docs/agents/antigravity.md](docs/agents/antigravity.md) |
| **OpenCode** | `tokenix install-hook --tool opencode` | MCP server only | [docs/agents/opencode.md](docs/agents/opencode.md) |
| **Any MCP client** | `tokenix mcp` | MCP server over stdio | [docs/agents/mcp.md](docs/agents/mcp.md) |
`tokenix install-hook --tool all` wires every agent except OpenCode. OpenCode's
registration lives in the repository, so you have to ask for it explicitly.
`tokenix remove-hook --tool <agent>` undoes the install.
---
## 🖥 Dashboard
Run bare `tokenix` for a terminal dashboard with twelve tabs and no flags. `←`/`→`
switch tabs, `↑`/`↓` move, `q` quits.
On a terminal, a report command opens its own tab: `tokenix doctor` lands on
Doctor, `tokenix scan-secrets` on Secrets. Piping, `--json`, `--statusline`,
`--format`, `--output` and `--no-tui` (or `TOKENIX_NO_TUI=1`) keep plain text
output. Agent-facing commands (`hook`, `run`, `mcp`, `read`, `pack`) never
open a UI.
| **Stats** | Version, per-agent hook status, index summary, one-key actions: index repo · install hooks · install binary on PATH |
| **Gain** | Tokens removed with a reduction bar, split by source and by command. `c` adds a ≈USD table at list input rates · `a` all projects · `r` refresh |
| **Usage** | Absolute token spend and ≈USD cost from agent transcripts. `s` cycles daily · model · 5-hour blocks · project · session |
| **Filters** | All 534 bundled filters by tool, with a live input → output preview and a per-filter token gauge |
| **Studio** | Record → preview → generate filters. Ranks the biggest unfiltered token sinks first (`⚠`), marks filtered commands (`✓`) and recordings (`●`) |
| **Secrets** | Credentials found in agent transcripts, grouped by rule, attributed to repo + branch. `v` reveal · `c` copy · `x` redact |
| **Egress** | External DNS/IP destinations in transcripts, checked against local reputation lists |
| **Graph** | Repo-wide symbol graph: god nodes, bottlenecks, blast-radius leaders |
| **Tokenmap** | The repository as a tree weighted by token count, heaviest paths first |
| **Discover** | Replays the current filters over past agent output: savings you could have had, plus commands no filter covers |
| **Audit** | MCP/tool weight of each agent's effective system prompt, plus always-on instruction files and skills |
| **Doctor** | Bundled-filter inventory, user/local filter config issues, active recording session |
<table>
<tr>
<td width="50%"><img src=".github/prints/stats.png" alt="Stats tab" /></td>
<td width="50%"><img src=".github/prints/gain.png" alt="Gain tab" /></td>
</tr>
<tr>
<td><img src=".github/prints/filters.png" alt="Filters tab" /></td>
<td><img src=".github/prints/secrets.png" alt="Secrets tab" /></td>
</tr>
</table>
---
## 📏 What we measure, and what we don't
**Every number below counts tokens removed from a payload before it reached the
model.** The measurements are reproducible. They are *not* a claim about your bill.
| Real sessions (7,807 hook calls, one developer's machine) | 475,360 → 169,175 | **67.4%** | `tokenix gain` |
| `tokenix read` outline vs the whole file, 31 real files (opt-in, not applied by the hook) | 346,892 → 58,154 | **83.2%** | `tokenix benchmark` |
| Outline + targeted symbol workflow | 55,020 → 17,384 | **68.4%** | `tokenix benchmark` |
| Command filters, verbose output | 1,891 → 369 | **80.5%** | `cargo test verbose_real_output -- --nocapture` |
| Command filters, full golden corpus (1,167 cases) | 46,804 → 27,728 | **40.8%** | `cargo test filters_deliver_aggregate_token_savings -- --nocapture` |
Baselines assume the agent would have read the whole file; there is no holdout group, so these are token measurements, not a causal effect. The benchmark rows use a small synthetic set from this repo and `benchmark/samples`, not a foreign codebase.
Golden filter cases reproduce their expected output byte for byte (**1,167/1,167**).
<details>
<summary><b>Why there is no dollar figure, and how to read these numbers</b></summary>
Published work measured *provider-billed* cost for hook-based compressors on Claude
Code. It found token reduction and cost reduction close to uncorrelated
([arXiv:2607.12161](https://arxiv.org/abs/2607.12161), 2,848 paired runs). One arm
removed 38.4% of tool-output tokens and billed **6.8% more**, with a per-task
correlation of r = 0.15. Prompt-cache traffic dominates a real bill, so tokens
removed from a fresh payload are mostly not the tokens you pay for. An independent
425-trial study of a competing tool measured a **+7.6%** cost increase while that
tool's analytics reported 96 million tokens saved.
We have not yet measured tokenix against provider-billed cost. Until then this
README reports tokens and calls them tokens. Cache-aware cost accounting is the top
roadmap item. [`docs/research/2026-08-token-economy.md`](docs/research/2026-08-token-economy.md)
has the full evidence review.
- **The 40.8% corpus figure is pessimistic on purpose.** Half the golden corpus is
failure-path cases a filter must pass through *unfiltered*, so errors are never
masked as success.
- **`tokenix benchmark`'s command arm reports about 26%** because its sample
commands print only 27–148 tokens each, which leaves little to compress.
- **Runs that saved nothing are in the denominator.** The 67.4% row was measured
before that rule existed, so it reads high. It will be re-measured before the
next release.
- **Your mileage depends on your codebase and your agent.** `tokenix gain` shows
what happened on your machine.
</details>
---
## 📖 Command reference
> [!TIP]
> Run `tokenix --help` for the full catalog with usage examples; `tokenix <command> --help` displays all flags and parameters for any specific command.
### 🤖 Commands agents use
| `tokenix read FILE` | Smart reader: outline for large files, full text for small ones (`--symbol`, `--lines`, `--mode full\|outline\|signatures\|diff\|density:X`) |
| `tokenix symbols QUERY` | Find indexed symbols by name or path (`--kind` filters by symbol type) |
| `tokenix callers SYMBOL` | Symbols that call or reference a symbol |
| `tokenix callees SYMBOL` | Symbols a symbol calls or references |
| `tokenix deps FILE` | File-level import dependencies (`--reverse`, `--transitive`, `--json`) |
| `tokenix impact SYMBOL` | Bidirectional impact graph (`--format html\|mermaid`) |
| `tokenix flow SYMBOL` | Forward call-flow trace (`--depth`, `--format text\|mermaid`) |
| `tokenix graph` | Repo-wide symbol-graph overview: god nodes, bottlenecks, blast-radius leaders, modules (`--format text\|dot\|json`, `--top N`) |
| `tokenix modules` | Functional modules found by community detection over the symbol graph (`--top`, `--json`) |
| `tokenix blast` | Blast radius of the current diff: changed symbols and everything that calls them (`--since REF`, `--depth`, `--json`) |
| `tokenix pack` | Budgeted repo pack for tools without hooks (`--mode/--profile`, `--changed`, `--token-map`) |
| `tokenix memory add\|list\|remove\|edit` | Save preferences (`--global` / `--project`); nothing injects them automatically, so read them with `memory list` or the MCP memory tools. Text that looks like a credential is refused |
| `tokenix retrieve KEY` | Print the exact original output behind a `[tokenix: ...]` marker |
### 🧑 Commands you run
| `tokenix` (no args) | Open the [dashboard](#-dashboard); piped or non-TTY falls back to help |
| `tokenix index [PATH]` | Index the repo at PATH (default `.`) |
| `tokenix export-index` / `import-index` | Write or load a shareable, secret-masked snapshot (`.tokenix/index.db.gz`) |
| `tokenix install-hook` / `remove-hook` | Install or remove agent hooks and instructions (default `--tool all`) |
| `tokenix install-binary` | Copy the running executable to a per-user bin dir and put it on PATH |
| `tokenix update` | Check for, install, or toggle automatic updates from GitHub releases (`--check`, `--auto`, `--enable-auto`, `--disable-auto`) |
| `tokenix doctor` | Diagnose the install: filter inventory, filter config and recording state |
| `tokenix gain` | Tokens removed, split by source (command filters vs Grep cap), plus session shape. Dollar estimates only with `--cost-estimate` / `--economics` |
| `tokenix discover` | Replay current filters over past agent output: recoverable savings plus uncovered commands (`--agent`, `--top`, `--json`) |
| `tokenix trust` / `untrust` | Approve (SHA-256 pinned) or revoke this repo's executable inputs (`--status`) |
| `tokenix usage` | Absolute token spend and ≈USD from agent transcripts (`daily\|weekly\|monthly\|session\|model\|project\|blocks`, `--all-projects`, `--statusline`, `--json`) |
| `tokenix stats` | Index statistics (files, chunks, tokens, age) |
| `tokenix tokenmap` | Directory tree weighted by token count, heaviest paths first (`--format html`) |
| `tokenix benchmark` | Reproducible token-reduction benchmark, vanilla vs tokenix (`--json`) |
| `tokenix filter list\|active\|generate\|record\|verify` | Browse, generate, record and golden-test output filters |
| `tokenix conversation-audit` | Scan local conversation histories for token-waste patterns (`--generate` prints ready-to-run filter commands) |
| `tokenix scan-secrets` | Scan agent transcripts for exposed credentials, attributed to repo + branch (`--group`, `--reveal`, `--json`) |
| `tokenix egress-audit` | Scan agent transcripts for external DNS/IP destinations, checked against local reputation lists |
| `tokenix artifacts list\|show` | Context artifacts from `.tokenix/artifacts.json` |
| `tokenix cycles` | Detect circular dependencies (Tarjan's SCC) |
| `tokenix rebuild-graph` | Rebuild graph tables from existing chunks from the stored chunks |
| `tokenix generate-ignores` | Write `.gitignore` entries for tokenix artifacts |
When you open `tokenix` in a terminal, it checks GitHub releases in the background
on a 24-hour cache and automatically installs a newer release by default.
The current command keeps running; the new version is used on the next launch.
Downloads use the release's SHA-256 checksums; a GitHub token, when set, is sent
only to the releases API, never to asset downloads.
Checks are skipped in CI, piped commands, hooks, and agent-facing commands. Set
`TOKENIX_AUTO_UPDATE=notify` to receive update hints without installing, or run
`tokenix update --disable-auto` to turn off automatic installation.
### ⚙ Internal (called by hooks, not by hand)
| `tokenix hook` | `PreToolUse` handler: caps unbounded Grep output and filters noisy commands (never touches `Read`) |
| `tokenix hook-post` | `PostToolUse` handler: redacts secrets and compresses tool output (agents whose install wires it, see the [agent guides](#-agent-guides)) |
| `tokenix run "CMD"` | Run a command and compress its output (`--shell`, `--path/-p`, `--raw`) |
| `tokenix mcp` | MCP server exposing context, read/search, graph and gain tools (`--profile slim\|full`) |
| `tokenix mcp-proxy -- CMD` | Wrap another stdio MCP server and compress only its `tools/call` text results |
<details>
<summary><b>Flags and environment variables</b></summary>
**Global:** `--no-tui` (or `TOKENIX_NO_TUI=1`)
prints plain text instead of opening the dashboard.
**Environment:**
| `TOKENIX_HOME` | Absolute path that replaces `~/.tokenix` (indexes, hook log, trust store, recall stash). Relative values are ignored. Useful for CI and throwaway runs |
| `TOKENIX_DISABLED=1` | Bypass the hook for one command |
| `TOKENIX_RAW=1` | Same as `tokenix run --raw` |
| `TOKENIX_MAX_OUTPUT_TOKENS` | Global output ceiling (default 8000, `0` disables) |
| `TOKENIX_BRANCH_AWARE=true` | One SQLite DB per git branch |
| `TOKENIX_GREP_HEAD_LIMIT` | `head_limit` injected into unbounded Grep (default 100, `0` disables) |
| `TOKENIX_AUTO_REFRESH=0` | Turn off the inline refresh before index commands |
| `TOKENIX_AUTO_REFRESH_MAX` | Dirty-file count above which the inline refresh is skipped (default 25) |
| `TOKENIX_DEDUP=0` · `TOKENIX_DEDUP_MIN_TOKENS` · `TOKENIX_DEDUP_TTL` | Cross-call dedup of identical output (defaults: on, 200 tokens, 3600 s; scoped to the project) |
| `TOKENIX_READ_DEDUP=0` · `TOKENIX_READ_DEDUP_TTL` · `TOKENIX_READ_DEDUP_MIN_TOKENS` | Re-read suppression (defaults: on, 900 s, 1500 tokens; scoped to the agent session) |
| `TOKENIX_TEE=0` | Do not tee failed-command output to `~/.tokenix/tee/` |
| `TOKENIX_AUTO_UPDATE=1\|0\|notify` | Control auto-update behavior (`1`/`install` auto-updates in background; `0` disables checks) |
| `TOKENIX_NO_UPDATE=1` | Disable all update checks and auto-updates |
**`tokenix index`:** `--force/-f`, `--cpu-profile <low|default|max>`,
`--jobs N`, `--if-stale`, `--path/-p`, `--no-low-priority` (indexing runs at below-normal priority by
default).
**`tokenix symbols`:** `--limit/-l` (20), `--kind/-k`, `--json`, `--path/-p`
**`tokenix deps`:** `--reverse`, `--transitive`, `--json`, `--path/-p`
**`tokenix impact`:** `--depth/-d` (2), `--limit/-l` (50),
`--format <text|html|mermaid|json>`, `--output/-o`, `--path/-p`
**`tokenix flow`:** `--depth/-d` (3), `--limit/-l` (50), `--format <text|mermaid>`
**`tokenix blast`:** `--since REF` (default `HEAD`, e.g. `origin/main`),
`--depth/-d` (2), `--limit/-l` (50), `--json`, `--path/-p`
**`tokenix modules`:** `--top` (12), `--json`, `--path/-p`
**`tokenix export-index` / `import-index`:** `--output/-o` and `--input/-i` (default
`.tokenix/index.db.gz`), `--force` on import to replace a newer local index. The
snapshot is a compacted copy of the index (legacy embedding tables are dropped).
Import refuses anything that is not a tokenix index, keeps the previous DB as
`*.pre-import.bak`, and keeps the snapshot's git fingerprint so `tokenix index`
only has to catch up on the diff.
**`tokenix pack`:** `--mode/--profile <plan|debug|audit|security|review>`,
`--budget N` (8000), `--format <markdown|xml|json>`, `--changed`, `--since REF`,
`--token-map`, `--output/-o`
**`tokenix install-hook` / `remove-hook`:**
`--tool <claude-code|copilot|codex|mcp|opencode|antigravity|all>` (default `all`),
`--local`
**`tokenix scan-secrets`:** `--agent <claude|gemini|copilot|antigravity|all>`,
`--filter <substr>`, `--group <none|value|rule|agent|file|repo>`, `--reveal`,
`--json`. Output is redacted by default, and the exit code is `1` when findings
exist. Rules are TOML `[[rules]]` (`id`, `pattern`, optional `capture` /
`min_entropy`). Bundled defaults live in `assets/secret-rules/`.
`~/.tokenix/secret-rules/*.toml` may add or override rules. A repo's
`<repo>/.tokenix/secret-rules/*.toml` loads only after `tokenix trust` and may only
*add*: an id that collides with a bundled rule is dropped with a warning.
**`tokenix egress-audit`:** `--agent`, `--filter`, `--group <host|rule|agent|file>`,
`--safe`, `--json`. Local reputation lists live in `~/.tokenix/safe-hosts.toml` and
`~/.tokenix/dangerous-hosts.toml`.
**`tokenix conversation-audit`:** `--agent`, `--min-chars N` (5000), `--limit N`
(30), `--json`, `--generate`
**`tokenix prompt-audit`:** `--agent`, `--json`, `--recommend`, `--profile-impact`
**`tokenix benchmark`:** `--json`, `--refresh-index`
</details>
---
## 🔧 Output filters
534 bundled filters, 1,167 golden cases. A filter matches a command and shapes its
output:
```toml
[filters.cargo-test]
match_command = '^cargo test'
strip_lines_matching = ['^\s*Compiling ', '^\s*Finished ']
priority_lines = ['(?i)^error', '^test result:']
tail_lines = 40
on_empty = "cargo test: all tests passed"
passthrough_when_emptied = true
[[tests.cargo-test]]
name = "all pass"
input = """..."""
expected = """cargo test: all tests passed"""
```
Filters resolve in this order: `<repo>/.tokenix/filters` (only after `tokenix
trust`) → `~/.tokenix/filters` → bundled. Every bundled filter ships **at least two
embedded golden cases**, and CI also enforces the rules below.
- **Failures are never masked.** A non-zero exit suppresses success messages, and a
filter that could empty a failure payload must keep the failure markers.
- **Never worse.** A filtered result never costs more bytes than the raw output.
- **Every regex compiles.** An invalid pattern would silently disable its rule;
CI fails on one in a bundled filter, and `tokenix doctor` lists them in yours.
- **stderr is opt-in per filter** (`filter_stderr = true`). The cargo filters set it,
because rustc writes every diagnostic there. `block_caps` then keeps each
warning's message and location and drops its snippet; errors keep theirs.
- **Line endings are preserved.** CRLF comes back as CRLF, so text the agent quotes
into an exact-match edit still matches the bytes on disk.
- **Compression happens at write time,** before output enters the conversation, so
an already-cached prompt prefix is never rewritten.
---
## 🧠 Supported languages
| **Symbol-aware chunking + call graph** (tree-sitter) | Rust, Python, JavaScript, TypeScript, Go, C, C++ |
| **Symbol-aware chunking** (line-based) | Visual Basic, SQL |
| **Indexed as text** (line chunks, searchable) | Shell (`.sh`, `.bash`), Markdown, TOML, plain text; JSON/YAML with `data_files = true` |
Anything else (Java, C#, Ruby, PHP, Kotlin, Swift, PowerShell, HTML, CSS …) is
**not indexed by default**. Opt in per project and those files are indexed as text:
```toml
[index]
extensions = ["java", "cs", "rb", "php", "kt", "swift", "ps1", "inc"]
[languages]
inc = "cpp" # parse an extension with a supported grammar: rust, python, typescript,
# javascript, go, cpp/c, vb, sql (anything else is chunked as text)
```
Files above `max_file_bytes` (1.5 MB) and binary files are skipped whole, never
truncated.
---
## 🏗 Architecture
```
┌─────────────┐ tool call ┌──────────────┐ pass (exit 0) / intercept (exit 2)
│ AI agent │ ───────────▶ │ tokenix hook │ ─────────────────────────────────▶
└─────────────┘ └──────┬───────┘
│
┌──────────────────┼──────────────────┐
▼ ▼ ▼
┌────────────┐ ┌────────────┐ ┌────────────┐
│ SQLite │ │ Symbol │ │ Output │
│ chunks │ │ graph + │ │ filters │
│ │ │ PageRank │ │ (TOML) │
└────────────┘ └────────────┘ └────────────┘
```
- **Storage:** one SQLite DB per project under `~/.tokenix/`: chunks and the
symbol graph. There is no search engine, no embeddings, no model and no
background daemon: the agent's own grep stays the way to find text. Indexes
built by older versions drop their vector and full-text tables on the next
`tokenix index`.
- **Fail-open contract:** the hook exits `0` (pass) on any error, a missing or
stale index, or even a panic. It exits `2` only to intercept, and never exits `1`.
---
## 🔒 Security
- **Your code stays local.** Indexing, search and interception make no network
calls. The network is used for:
- the self-updater, which talks to this repo's GitHub releases only;
- `tokenix filter generate`, which sends your recorded (redacted) output to an AI
CLI **you** have installed, and asks before it re-runs any command.
- **Repositories are untrusted by default.** A repo's `.tokenix/filters`,
`.tokenix/secret-rules`, `.tokenix/egress-rules`, `.mcp.json`, `opencode.json` and
`.vscode/mcp.json` are ignored until `tokenix trust` pins their SHA-256, and any
edit revokes that trust. A cloned repo cannot change what your agent sees, blind
the secret scanner or make `prompt-audit` spawn a server of its choosing.
User-scoped configs (`~/.claude.json`, `~/.codex/`) are yours and are not gated.
- **Credentials are masked before anything is saved.** The hook log, the failure
tee, recordings and index snapshots go through one redactor, memory refuses
notes that look like a credential, and `~/.tokenix` is owner-only (`0600`). The recall stash is the deliberate exception:
`tokenix retrieve` must return the exact original bytes, so it relies on file
permissions alone.
- **Secrets are not indexed.** `.env`, `.pem`, keystores and similar files are
excluded, and `tokenix scan-secrets` redacts its output by default.
- **Supply chain:** releases publish `sha256sums.txt` and SLSA provenance.
Workflows are SHA-pinned and checked by `cargo-deny`, `zizmor` and OpenSSF
Scorecard. See [SECURITY.md](SECURITY.md).
---
## ❓ Troubleshooting
| `No index found` | Run `tokenix index .` in the repository root. The nearest indexed directory counts as the project root, even when a parent directory has a `package.json`. |
| The agent still reads whole files | By design: tokenix never intercepts `Read`. The agent can call `tokenix read` or `tokenix symbols` itself. |
| A symbol I just wrote is missing | It should not: edits are re-chunked before each command. If more than 25 files changed, run `tokenix index` (or raise `TOKENIX_AUTO_REFRESH_MAX`). |
| A repo filter is ignored | Run `tokenix trust --status`. Repo filters need `tokenix trust`, and an edit revokes it. |
| Output was cut and I need all of it | Run `tokenix retrieve <key>` from the `[tokenix: ...]` marker, or `tokenix run --raw`. |
| First index is slow or the machine stalls | Use `tokenix index --cpu-profile low`. |
| Offline or air-gapped | Nothing to download: the only network use is the self-updater (`TOKENIX_NO_UPDATE=1` disables it). |
| Remove everything | `tokenix remove-hook --tool all`, then delete `~/.tokenix` (or your `TOKENIX_HOME`). |
---
## 🤝 Contributing
See [CONTRIBUTING.md](CONTRIBUTING.md). `./scripts/verify.sh` runs every CI gate
locally, in CI order: fmt, clippy, unit, golden and end-to-end tests, release build
and the homologation scripts. It never
touches your real `~/.tokenix`. New filters need at least two golden cases, and
[`AGENTS.md`](AGENTS.md) lists the engine rules a change must not break.
## 📄 License
MIT. See [LICENSE](LICENSE).