β‘ 10-Second Quickstart
Add Tokenectomy to your coding agent with zero configuration. Works immediately via npx (no Rust toolchain required):
1. Cursor IDE
Add to .cursor/mcp.json in your workspace root (or global ~/.cursor/mcp.json):
2. Claude Desktop
Add to your claude_desktop_config.json:
3. Or Pipe Directly in Your Terminal
|
π The Problem: Why Your AI Hits Rate Limits & Hallucinates
When your app crashes during development (Next.js, Express, FastAPI, Tokio), the runtime dumps hundreds of lines of third-party plumbing from node_modules or site-packages.
When you paste that raw crash dump into Cursor or Claude:
- Eats Your 5-Hour Rate Limit: A single Express/Prisma error can dump 5,000 to 45,000 tokens of third-party library code you never wrote. A few crash loops easily burn your session limit.
- Triggers AI Hallucinations: Claude gets lost in framework internals (
node_modules/express/lib/router/layer.jsorstarlette/routing.py) and tries to edit library files instead of your actual application code. - Leaks Secrets & Credentials: Connection strings with raw database passwords, JWT bearer tokens, and cloud keys embedded in error traces get forwarded to external model servers.
Raw Terminal Crash (45,820 tokens + Leaked Secrets)
β
βΌ <0.2ms Local Rust DFA Engine
[Redact Passwords & Keys] βββΊ [Strip Third-Party Framework Frames] βββΊ [Isolate Root Cause]
β
βΌ
Clean Context (118 tokens β’ Zero Secrets β’ Sub-millisecond)
π Before & After Comparison
β Without Tokenectomy: AI Hallucinates & Burns 45,000 Tokens
TypeError: Cannot read properties of undefined (reading 'digest')
at Object.<anon> (/node_modules/next/bundle5.js:142:31)
at __webpack_require__ (/node_modules/next/bundle5.js:198:12)
at Object.execute (/node_modules/next/dev-server.js:412:19)
at processTicksAndRejections (task_queues:95:5)
Database connection failed: postgresql://admin:super_secret_password@db.prod.internal:5432/primary
API key leaked: sk-ant-api03-abcdef1234567890abcdef1234567890
[... 480 internal dependency frames flooding your context window ...]
What Claude does: Tries to understand
bundle5.js, suggests modifying your webpack bundle or adjusting Next.js internals, and eats a massive chunk of your context window. Leaks your database credentials to external logs.
β With Tokenectomy: Clean Context & Instant Fix
// [Tokenectomy Surgery: 480 framework frames pruned (99.7%)]
// Source: src/components/Header.tsx:42
42 | const user = useSession( ;
| ^ Expected ')'
π‘οΈ [CONNECTION_STRING_REDACTED]
π‘οΈ [REDACTED_ANTHROPIC_KEY]
What Claude does: Instantly identifies that line 42 in
Header.tsxis missing a closing parenthesis). Applies the exact 1-line fix in 2 seconds. Credentials redacted before transmission. 99.7% token reduction.
π¬ Verifiable Benchmarks
All performance claims are hardware-grounded and independently reproducible on physical hardware (measured on 10-Core Intel Core i5-1235U @ 15W running Arch Linux, Kernel 6.13):
Hardware Dependency Notice: Performance is hardware-dependent; reported throughput represents measured results on the specified test hardware (10-Core Intel Core i5-1235U @ 15W TDP). Throughput scales with higher TDP desktop/server CPUs and faster memory buses. Developers are encouraged to independently audit performance using the reproduction command below.
$ cargo test --release --test stress_benchmark -- --nocapture
=====================================================================================
π§ͺ TOKENECTOMY OSS VERIFIABLE HEAVY STRESS BENCHMARK (100% REPRODUCIBLE IN OSS)
Hardware: 10-Core / 12-Thread Intel Core i5-1235U | OS: Arch Linux | Kernel Telemetry Active
Initial Baseline Process Memory (VmRSS): 3.45 MB
=====================================================================================
π₯ [TEST 1/3] QUARTER-MILLION LINES LOG REDACTION TORTURE (250,000 LINES / 25MB+ BUFFER)
βββ Buffer Size: 24.44 MB (250000 lines)
βββ Redaction Latency: 471.05ms (51.9 MB/sec)
βββ Line Throughput: 530,735 lines/sec
βββ Peak Memory (VmRSS): 76.05 MB (Delta: +72.60 MB)
βββ Status: β
PASSED (100% of 250,000 lines sanitized, zero memory balloon)
π₯ [TEST 2/3] REDOS CATASTROPHIC BACKTRACKING TORTURE (50,000 CHARS PAYLOAD)
βββ Attack Payload Size: 50,082 characters
βββ Execution Latency: 1.165 ms
βββ Status: β
PASSED (Linear O(N) evaluation, ReDoS-resistant on tested payloads)
π₯ [TEST 3/3] HIGH-CONCURRENCY TORTURE (100 PARALLEL OS THREADS)
βββ Thread Concurrency: 100 concurrent OS threads
βββ Successful Operations: 100/100 (100.0%)
βββ Total Elapsed: 11.31ms
βββ Concurrency Throughput: 17,688 ops/sec
βββ Final VmRSS: 78.99 MB
βββ Status: β
PASSED (Zero race condition, zero deadlock)
=====================================================================================
π TOKENECTOMY OSS STRESS BENCHMARK: 3/3 PASSED (100% GREEN)
Total Suite Duration: 580.83ms
Bounded Final VmRSS: 78.99 MB
=====================================================================================
test test_oss_heavy_stress_benchmark ... ok
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.58s
| Benchmark Target | Workload Under Test | Verified Measurement | Result |
|---|---|---|---|
| Log Redaction Throughput | 250,000 lines (24.44 MB) enterprise dump with API keys & connection URIs | 530,735 lines/sec (471.0 ms, 51.9 MB/s) | Pass |
| ReDoS Resilience | 50,000-character pathological backtracking regex payload | 1.16 ms (Deterministic Linear $O(N)$ DFA Evaluation) | Pass |
| Thread Concurrency | 100 concurrent OS threads executing simultaneous redaction | 17,688 ops/sec (100/100 completed in 11.31 ms) | Pass |
| Memory Footprint | Peak Resident Memory during 250k-line continuous stress test | 76.05 MB VmRSS via /proc/self/status |
Pass |
| Release Test Suite | Full integration test matrix across extractors, filters, and analyzers | 57 / 57 Verified Green (Zero panics, zero leaks) | Pass |
π‘οΈ Automated Redaction & Secret Sanitization Benchmark
Automated evaluation across 12 polyglot crash traces (Rust, Python, TypeScript, Go, YAML) containing 22 ground-truth credentials and clean negative controls. Evaluated head-to-head against Gitleaks v8.30.1.
1. Per-Category Precision, Recall & F1-Score
| Secret Category | Ground Truth | Razor Recall | Razor F1 | Gitleaks Recall | Gitleaks F1 | Sanitization Advantage |
|---|---|---|---|---|---|---|
Anthropic Claude API Key (sk-ant-...) |
1 | 100.0% | 100.0% | 0.0% | 0.0% | +100% Recall (M2M zero-leak) |
AWS Access Key ID (AKIA...) |
1 | 100.0% | 100.0% | 0.0% | 0.0% | +100% Recall (M2M zero-leak) |
| AWS Secret Access Key | 1 | 100.0% | 100.0% | 0.0% | 0.0% | +100% Recall (M2M zero-leak) |
| Database URI (PostgreSQL, MySQL, Redis, Mongo) | 4 | 100.0% | 100.0% | 0.0% | 0.0% | +100% Recall (M2M zero-leak) |
| Generic Passwords / Auth Secrets (YAML/JSON) | 3 | 100.0% | 100.0% | 0.0% | 0.0% | +100% Recall (M2M zero-leak) |
GitHub Personal Access Token (ghp_...) |
1 | 100.0% | 100.0% | 0.0% | 0.0% | +100% Recall (M2M zero-leak) |
GitLab Personal Access Token (glpat-...) |
1 | 100.0% | 100.0% | 100.0% | 100.0% | Parity (100% caught) |
HuggingFace API Token (hf_...) |
1 | 100.0% | 100.0% | 0.0% | 0.0% | +100% Recall (M2M zero-leak) |
| JSON Web Token (RFC 7519 / Truncated) | 2 | 100.0% | 100.0% | 100.0% | 100.0% | Parity (100% caught) |
npm Registry Access Token (npm_...) |
1 | 100.0% | 100.0% | 0.0% | 0.0% | +100% Recall (M2M zero-leak) |
OpenAI API Key (sk-..., sk-proj-...) |
1 | 100.0% | 100.0% | 100.0% | 100.0% | Parity (100% caught) |
| PEM Private RSA Key Block | 1 | 100.0% | 100.0% | 100.0% | 100.0% | Parity (100% caught) |
PyPI Package Upload Token (pypi-AgEI...) |
1 | 100.0% | 100.0% | 100.0% | 100.0% | Parity (100% caught) |
SendGrid API Key (SG...) |
1 | 100.0% | 100.0% | 0.0% | 0.0% | +100% Recall (M2M zero-leak) |
Slack Bot/User Token (xoxb-...) |
1 | 100.0% | 100.0% | 100.0% | 100.0% | Parity (100% caught) |
Stripe Live/Test Secret Key (sk_live_...) |
1 | 100.0% | 100.0% | 100.0% | 100.0% | Parity (100% caught) |
2. Head-to-Head Performance & Architectural Summary
| Dimension | Tokenectomy Razor (--scrub) |
Gitleaks v8.30.1 | Architectural Rationale |
|---|---|---|---|
| Overall Secret Recall | 100.0% (22/22) | 36.4% (8/22) | Razor captures unquoted URIs, DB ports & AI keys missed by diff rules |
| Overall Precision | 100.0% (0 False Positives) | 88.9% | Zero false triggers on compiler errors & minified traces |
| Overall F1-Score | 100.0% | 51.6% | Comprehensive coverage engineered specifically for crash context |
| Execution Engine | Zero-allocation Rust DFA ($O(N)$) | Go regex scanner + Git tree crawler | Sub-millisecond latency for agent streaming backtraces |
| ReDoS Resilience | Deterministic Linear Time ($O(N)$) | Engine dependent | Non-backtracking DFA regex prevents catastrophic backtracking on tested dumps |
| Sanitization Action | Inline token redaction ([KEY_REDACTED]) |
Warning log only (No scrub) | Directly sanitizes text before ingestion by LLM cortex |
3. Token Reduction & LLM Context Savings (tiktoken cl100k_base)
| Metric | Measured Value | Operational Impact for AI Coding Agents |
|---|---|---|
| Mean Token Reduction | 41.67% | Consistently shrinks raw crash trace token footprint |
| Median Reduction (P50) | 42.95% | Typical credential and connection dump reduction |
| 90th Percentile (P90) | 61.42% | Eliminates long multi-line keys and credentials |
| Min / Max Spread | 0.00% β 81.36% | 0% on clean negative controls (zero distortion), up to 81.4% on leaks |
| Total Tokens Preserved / Saved | 920 tokens (44.02% net) | Prevents context window saturation and reduces LLM billing |
π Quick Start
1. Model Context Protocol (MCP) Setup
Tokenectomy Razor operates natively over JSON-RPC 2.0 stdio, compliant with the official Model Context Protocol specification.
Cursor Composer
Add to .cursor/mcp.json in your workspace root:
Claude Desktop
Add to claude_desktop_config.json:
- macOS:
~/Library/Application Support/Claude/claude_desktop_config.json - Windows:
%APPDATA%\Claude\claude_desktop_config.json - Linux:
~/.config/Claude/claude_desktop_config.json
Windsurf (Codeium)
Add to ~/.codeium/windsurf/mcp_config.json:
VS Code (Cline / Roo Code)
In Cline or Roo Code settings (cline_mcp_settings.json):
Google Antigravity CLI
2. Standalone CLI & Terminal Piping
When debugging or piping terminal output directly:
# Pipe terminal test failures through the surgical redactor:
|
# Sanitize a specific raw log file:
# Offline local air-gapped mode (zero external network calls):
|
3. AI Gateway Reverse Proxy (--proxy)
Tokenectomy Razor can operate as a high-throughput local HTTP reverse proxy on 127.0.0.1:8080. It intercepts outbound prompt streams, performs real-time token excision and credential sanitization, and forwards clean requests upstream to OpenAI, Anthropic, or Ollama.
# Start local gateway proxy forwarding to OpenAI:
# Start local gateway forwarding to Ollama:
Point any standard SDK client to the local proxy:
FinOps Economics Dashboard
Open http://127.0.0.1:8080/dashboard in any browser to monitor real-time token savings, dollar savings (blended LLM pricing), total requests, and active security redactions.
π οΈ Exposed MCP Tools
Tokenectomy Razor complies with Glama Grade A Tool Definition Quality Score (TDQS) with explicit parameter boundaries:
| Tool Name | Capability Description |
|---|---|
get_error_context |
Performs trace surgery on error dumps, removes framework noise, redacts credentials, and extracts relevant local source context bounded to the workspace. |
analyze_code |
Performs static AST code analysis to detect resource leaks, unclosed handles, and syntax vulnerabilities with bounded execution limits and precise LSP UTF-16 coordinates. |
apply_code_patch |
Applies atomic file modifications with post-write language syntax verification (cargo check, py_compile, node --check) and automated rollback on failure. |
search_stack_overflow |
Queries Stack Exchange API for relevant error signatures using sanitized, redacted search terms. |
audit_context_health |
Audits raw logs, traces, or prompt payloads for token bloat, framework noise, and credentials. Returns M2M telemetry, savings metrics, and context health grades. |
π Supported Polyglot Ecosystems
| Language | Frameworks Supported | Excluded Framework Internals |
|---|---|---|
| Rust | Tokio, Actix-web, Axum | .cargo/registry, .rustup, target/debug/build |
| TypeScript / JS | Next.js, Express, NestJS, Vite | node_modules, .next, dist, webpack internals |
| Python | Django, FastAPI, Flask, PyTorch | site-packages, dist-packages, venv, __pycache__ |
| Golang | Gin, Fiber, Stdlib Panics | go/src (stdlib), go/pkg/mod, vendor |
| Java / Kotlin | Spring Boot 3, Tomcat, Netty | .m2/repository, .gradle/caches, internal bytecode |
| C / C++ | AddressSanitizer, GDB / LLDB | /usr/include, /usr/lib, vcpkg_installed |
| C# (.NET) | ASP.NET Core, .NET Runtime | System.Private.CoreLib, Microsoft.AspNetCore |
| Ruby on Rails | Rails, Sinatra, Bundler | /gems/, ruby/gems, internal rack handlers |
| PHP | Laravel, Symfony | vendor/composer, vendor/symfony |
π¦ Installation Options
Method 1: Instant via npx (Zero Toolchain Setup)
Method 2: Cargo (crates.io)
Method 3: Precompiled Standalone Binaries
Zero-dependency, standalone release binaries available on GitHub Releases:
- Linux:
x86_64-unknown-linux-gnu,x86_64-unknown-linux-musl,aarch64-unknown-linux-gnu - macOS:
aarch64-apple-darwin(Apple Silicon M1/M2/M3/M4),x86_64-apple-darwin(Intel) - Windows:
x86_64-pc-windows-msvc.exe
Method 4: Multi-Arch Docker Container (GHCR)
βοΈ Edition Comparison
| Capability | Razor (Community OSS) | Sentinel (Commercial Tier) |
|---|---|---|
| Polyglot Stack Trace Surgery | Yes (4 Languages) | Yes (All 7 Languages) |
| O(N) ReDoS-Safe Secret Redaction | Yes | Yes |
| JSON-RPC 2.0 MCP Server | Yes | Yes |
AI Gateway Reverse Proxy (--proxy) |
Yes | Yes |
| SHA-256 Idempotency Cache (24h TTL) | Yes | Yes |
| FinOps Metrics Dashboard | Yes | Yes |
| Tree-sitter AST Syntax Healing | β | Yes |
| Anti-Hallucination Scope Guard | β | Yes |
| Automated Test Rollback (0 Dirty Diff) | β | Yes |
| Multi-File Atomic Transactions | β | Yes |
Time Machine Undo Engine (--undo) |
β | Yes |
| Autonomous Healing State Machine | β | Yes |
Need Enterprise AST Self-Healing? Explore the Sentinel Tier
πΊοΈ Roadmap & Milestones
| Milestone / Capability | Status | Target |
|---|---|---|
| Core Polyglot Log Surgery & $O(N)$ ReDoS Redaction | β Complete | v1.0.0 |
AI Gateway Reverse Proxy (--proxy) & Idempotency Cache |
β Complete | v1.1.0 |
| Multi-arch Docker & GitHub Actions Marketplace Action | β Complete | v1.1.3 |
Static AST Analysis Engine (analyze_code) & UTF-16 LSP |
β Complete | v1.1.5 |
| Glama.ai Tool Definition Quality Score (TDQS Grade A) | β Complete | v1.1.5 |
| Precompiled Standalone Binaries (Linux, macOS, Windows) | β Complete | v1.1.6 |
Official MCP Registry Listing (io.github.Tokenectomy-Labs/razor) |
β Complete | v1.1.7 |
mcpservers.org Directory Listing |
β Complete | v1.1.7 |
| Java/Kotlin (Spring Boot 3) & C/C++ (ASan) Extractors | β Complete | v1.2.0 |
| C# (.NET) & Ruby on Rails Deep Stack Surgery | β Complete | v1.2.2 |
User-defined custom redaction & noise rules (~/.tokenectomy.toml) |
β Complete | v1.2.2 |
Autonomous Context Health Audit (audit_context_health) & M2M Advisory |
β Complete | v1.2.2 |
| Automated Redaction Benchmark & CI Gate | β Complete | v1.2.2 |
Declarative Advisory M2M Control Plane ([:TOKENECTOMY:M2M_CONTROL_PLANE:v1.3.0]) |
β Complete | v1.3.0 |
Interactive Multi-Strategy Budgeting (aggressive, conservative, lossless_compact) |
β Complete | v1.2.3 |
| GitHub Actions OIDC Official Registry Publishing Gate | β Complete | v1.2.3 |
awesome-mcp-servers Community Catalog Listing |
β Complete | v1.2.4 |
Inline Dropped Frame Identities ([DROPPED_FRAMES: ...]) & Anti-Silent Truncation Audit |
β Complete | v1.3.1 |
Content-Addressable Raw Log Cache & Verification Hash (--diff-verify) |
β Complete | v1.3.1 |
| Native VS Code & JetBrains companion extensions | π Planned | v1.4.0 |
| Server-Sent Events (SSE) remote MCP transport | π Planned | v1.4.0 |
π Security & Invariants
- Zero-Knowledge Architecture: All parsing, filtering, and secret redaction execute on physical local hardware. No logs are ever transmitted to third-party telemetry servers.
- Deterministic Linear-Time Pattern Matching: All pattern matchers utilize finite automaton evaluation (Rust non-backtracking DFA regex engine and Aho-Corasick) providing deterministic $O(N)$ linear time guarantees on tested adversarial inputs.
- Path Traversal Boundary Isolation: File operations are strictly locked within the active workspace root (
CWD). Path traversals (../) and unauthorized symlinks are blocked. - Safe Rust Implementation: Core execution paths enforce safe Rust memory guarantees with bounded stream readers (
.take()) preventing resource exhaustion.
For vulnerability disclosures, please review our Security Policy.
βοΈ Legal & Downstream Fork Disclaimer
Tokenectomy Razor is provided strictly for lawful developer productivity, observability, log surgery, and defensive credential redaction. Any downstream forks, clones, redistributions, or private deployments operate completely independently of the original authors. Tokenectomy Labs and its maintainers assume zero liability for unlawful, malicious, or unauthorized actions committed by third parties using this codebase or derivatives thereof. All downstream operators bear 100% individual responsibility for compliance with local and international cybersecurity laws. See DISCLAIMER.md for full legal terms.
π€ Community & Resources
- π Official Website
- π Documentation
- ποΈ System Architecture
- π Changelog
- π€ Contributing Guidelines
- π Security Policy
- βοΈ Disclaimer & Liability