tokenectomy 1.1.5

Tokenectomy Razor πŸ—‘οΈ β€” Autonomous M2M Model Context Protocol (MCP) server & log surgery engine for AI coding agents (Claude, Cursor, Cline, Antigravity)
Documentation

Tokenectomy Razor

Fast, deterministic log surgery and secret redaction for AI coding agents β€” purge 90%+ framework noise, redact credentials with O(N) ReDoS immunity, sub-millisecond latency. Written in safe Rust.

Tokenectomy (noun): token + -ectomy (surgical removal) β€” the precise excision of wasteful tokens from LLM context windows.

High-Performance Log Surgery & Secret Redaction Engine for AI Coding Agents


πŸ“‹ Table of Contents


What It Does

Tokenectomy Razor is an autonomous, machine-to-machine (M2M) Model Context Protocol (MCP) server and stream processing engine written in safe Rust. It intercepts error logs from AI agents, strips 90%+ of framework noise, automatically redacts secrets (JWTs, API keys, database credentials), and caches sanitized contexts with a 24-hour TTLβ€”all without sending raw data to external services.

In 30 Seconds

The Problem:

  • AI agents waste tokens on framework noise (node_modules, site-packages, .cargo/registry)
  • Sensitive credentials accidentally leak into LLM logs (AWS keys, database URLs, API tokens)
  • Repeated identical errors cost money for every retry

The Solution:

Raw Error Log (38K tokens + secrets)
    ↓
[Redact secrets locally] β†’ [Filter framework frames] β†’ [Extract user code]
    ↓
Sanitized Context (2K tokens, no secrets) β†’ Safe to send to LLM

Real Example

Before:

$ cat error.log | head -20
Error in /home/user/.cargo/registry/src-xxx/tokio-1.35/src/runtime/mod.rs:12345
  at /home/user/.cargo/registry/src-yyy/serde/src/lib.rs:456
  Database connection failed: postgresql://admin:secretpass@db.example.com:5432/mydb
  JWT Auth token: eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIn0...
  [... 500+ more framework frames ...]

After:

$ cat error.log | razor --scrub
Error in /home/user/src/main.rs:42
  at /home/user/src/utils.rs:18
  Database connection failed: [CONNECTION_STRING_REDACTED]
  JWT Auth token: [JWT_REDACTED]

Benefits:

  • βœ… 95% smaller context (2K vs 38K tokens) β†’ Save money on LLM API calls
  • βœ… Zero secrets in logs β†’ Sleep better at night
  • βœ… Identical errors cached β†’ Second retry costs $0

Technical Highlights

                  β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
   Agent Error     β”‚              TOKENECTOMY RAZOR               β”‚    Sanitized Context
   Dump (38K toks) β”‚  - Polyglot Stack Frame Filter               β”‚ ──►  (2K toks) ──► LLM
  ────────────────►│  - Deterministic Secret Redactor (O(N))      β”‚
                   β”‚  - SHA-256 Idempotency Cache (24h TTL)       β”‚
                   β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
  • Polyglot Trace Surgery: In-memory parsing across Rust, Python, TypeScript/JavaScript, and Go. Filters noisy dependency frames and isolates user-written code only. (Java, C/C++, PHP support coming in v1.2)
  • AI Gateway Reverse Proxy (--proxy): Transparently intercepts prompt streams on 127.0.0.1:8080, performing real-time token excision and credential sanitization before upstream forwarding to OpenAI, Anthropic, or Ollama.
  • Zero-Knowledge Secret Redaction: Linear-time deterministic regex engine strips JWTs, API tokens, cloud access keys, connection strings, and private keys prior to network transmission. All processing happens locally.
  • SHA-256 Idempotency Cache: Stores deterministic responses with a 24-hour TTL. Repeated CI/CD or agent loop failures incur zero upstream API cost.
  • Path Traversal Containment: All MCP filesystem access is canonicalized and locked to the workspace root boundary (CWD). No ../ escapes or symlink breakouts.
  • M2M Protocol Compliance: Native JSON-RPC 2.0 stdio server compliant with the official Model Context Protocol specification.

Verifiable Benchmarks

Performance metrics are hardware-grounded and reproducible via standalone benchmark suites:

Benchmark Target Workload Under Test Verified Measurement Result
High-Volume Log Redaction 250,000 lines (24.44 MB) enterprise dump containing API keys and connection URIs 333.49 ms (73.3 MB/sec, 749,652 lines/sec) Pass
ReDoS Resistance 50,000-character pathological backtracking string 1.44 ms (Linear $O(N)$ evaluation) Pass
Thread Concurrency 100 concurrent OS threads executing simultaneous redaction and extraction 100/100 completed in 27.35 ms (7,312 ops/sec) Pass
Kernel Memory Footprint Peak Resident Memory during 250,000-line continuous stress test 76.24 MB VmRSS via /proc/self/status Pass

Understanding the Benchmarks

Metric Why It Matters What To Expect
73.3 MB/sec redaction throughput Most logs are <5MB; you'll redact them in milliseconds <10ms for typical CI logs
1.44ms ReDoS immunity Prevents malicious log payloads from DoS'ing your system Safe to use in production with untrusted input
76.24 MB peak memory Suitable for constrained CI/CD runners (GitHub Actions, GitLab) Fits within 256MB limits comfortably
7,312 ops/sec concurrent Multiple AI agents querying simultaneously 100 concurrent requests handled safely

Reproduce locally:

cargo test --release --test stress_benchmark -- --nocapture

Installation

Method 1: Instant via npx (Recommended for MCP Clients)

No Rust toolchain, native compilation, or manual path setup required:

npx -y tokenectomy-razor --mcp

Or install globally via npm:

npm install -g tokenectomy-razor

Method 2: Cargo (crates.io)

cargo install tokenectomy

Method 3: Build from Source

git clone https://github.com/Tokenectomy-Labs/Tokenectomy.git
cd Tokenectomy
cargo build --release
sudo cp target/release/razor /usr/local/bin/razor

Method 4: Multi-Arch Container (GHCR)

docker pull ghcr.io/tokenectomy-labs/razor:latest
docker run -it ghcr.io/tokenectomy-labs/razor:latest --help

Model Context Protocol (MCP) Integration

Configure Tokenectomy Razor as an autonomous background server across major AI agent environments:

Claude Desktop

Add to ~/Library/Application Support/Claude/claude_desktop_config.json (macOS) or %APPDATA%\Claude\claude_desktop_config.json (Windows):

{
  "mcpServers": {
    "tokenectomy": {
      "command": "npx",
      "args": ["-y", "tokenectomy-razor", "--mcp"]
    }
  }
}

Cursor

Add to .cursor/mcp.json in your project root:

{
  "mcpServers": {
    "tokenectomy": {
      "command": "npx",
      "args": ["-y", "tokenectomy-razor", "--mcp"]
    }
  }
}

Cline / Roo Code / Windsurf / VS Code

Add to your client configuration (cline_mcp_settings.json or settings.json):

{
  "mcpServers": {
    "tokenectomy": {
      "command": "npx",
      "args": ["-y", "tokenectomy-razor", "--mcp"]
    }
  }
}

Google Antigravity CLI

agy mcp add tokenectomy-razor -- npx -y tokenectomy-razor --mcp

Exposed MCP Tools

Tool Name Capability Description
get_error_context Performs trace surgery on error dumps, removes framework noise, redacts credentials, and extracts relevant local source context bounded to the workspace.
search_stack_overflow Queries Stack Exchange API for relevant error signatures using sanitized search terms.
apply_code_patch Applies atomic file modifications with post-write language syntax verification (cargo check, py_compile, node --check) and automated rollback on validation failure.
analyze_code Performs static AST code analysis to detect resource leaks, security vulnerabilities, and code defects with bounded execution limits and precise LSP UTF-16 coordinates.

Quick Start

I use Claude Desktop

# 1. Add to claude_desktop_config.json (see MCP Integration section above)
# 2. When Claude encounters errors, it automatically uses "get_error_context" tool
# 3. Errors stay sanitized without any additional setup

I use GitHub Actions

# Add to your workflow (.github/workflows/build.yml)
- name: Sanitize Build Failure Log
  if: failure()
  uses: Tokenectomy-Labs/Tokenectomy@v1
  with:
    log-file: 'build.log'
    output-file: 'sanitized.log'

# Now you can safely share sanitized.log without leak concerns

Parameters:

Parameter Type Default Description
log-file String '' Path to raw error log file to process
log-content String '' Direct string content if file is not specified
output-file String tokenectomy-sanitized.log Path for scrubbed output file
version String v1.1.3 Binary release target version

I want max privacy (air-gapped environment)

# All redaction happens locallyβ€”no network calls except to your LLM
cargo install tokenectomy

# Process logs without any cloud services
echo $ERROR_LOG | razor --scrub --local-only

# Or from a file:
razor --scrub --file /var/log/app/error.log > sanitized.log

Advanced Usage

Standalone CLI

In addition to M2M agent mode, Razor provides CLI commands for terminal piping and local shell scripting:

# Scrub framework frames and output clean log
npm test 2>&1 | razor --scrub > sanitized.log

# Sanitize a specific log file
razor --scrub --file /var/log/app/error.log > sanitized.log

# CLI diagnosis with specific AI provider
razor --file error.log --provider openai
razor --file error.log --provider anthropic
razor --file error.log --local-only

AI Gateway Reverse Proxy Mode

Tokenectomy Razor can operate as a transparent local HTTP reverse proxy. It sits between client applications and upstream LLM providers (OpenAI, Anthropic, Ollama, OpenRouter), performing real-time token excision and credential sanitization before upstream forwarding.

Local Development (Default Loopback):

# Forward to OpenAI
razor --proxy --proxy-bind 127.0.0.1:8080 --upstream-url https://api.openai.com/v1

# Forward to local Ollama instance
razor --proxy --proxy-bind 127.0.0.1:8080 --upstream-url http://127.0.0.1:11434/v1

Point any standard SDK or IDE client to the local proxy:

export OPENAI_BASE_URL="http://127.0.0.1:8080/v1"
# Now all API calls are automatically sanitized

Production Proxy Hardening

Binding to external interfaces (0.0.0.0) requires explicit token authorization:

razor --proxy --proxy-bind 0.0.0.0:8080 --upstream-url https://api.openai.com/v1 --allow-remote --proxy-token "YOUR_SECURE_TOKEN"

Resource limits enforced: MAX_HEADER_SIZE (64 KB), MAX_BODY_SIZE (10 MB), client/upstream timeouts (30s / 60s), and a 128-connection concurrency cap.

Configuration

Configuration values can be set via ~/.tokenectomy.toml:

default_provider = "openai"  # openai | anthropic | ollama | mock
openai_api_key = "sk-..."
anthropic_api_key = "sk-ant-..."
ollama_base_url = "http://localhost:11434"
context_lines = 10
max_context_chars = 10000

Supported Ecosystems

Language Primary Frameworks Excluded Framework Paths
Rust Tokio, Actix-web, Axum .cargo/registry, .rustup, target/debug/build
Python Django, FastAPI, Flask, PyTorch site-packages, dist-packages, venv, __pycache__
TypeScript / JavaScript Next.js, Express, NestJS, Vite node_modules, .next, dist, webpack internals
Golang Gin, Fiber, Stdlib panics go/src (stdlib), go/pkg/mod, vendor
Java / Kotlin Spring Boot, Quarkus, Gradle .m2/repository, .gradle/caches, framework internals
C / C++ GDB Backtraces, AddressSanitizer /usr/include, /usr/lib, vcpkg_installed
PHP Laravel, Symfony vendor/composer, vendor/symfony, vendor/laravel

Note: Currently shipped with robust extractors for Rust, Python, TypeScript/JavaScript, and Go. Java/Kotlin, C/C++, and PHP support is coming in v1.2. See #1 for progress tracking.


How Tokenectomy Compares

Feature Tokenectomy Splunk Log Obfuscation Datadog Logs git-secrets
Instant setup (no agent install) βœ… ❌ ❌ βœ…
Works with AI agents (MCP) βœ… ❌ ❌ ❌
Local-only processing βœ… ❌ ❌ βœ…
Polyglot stack traces βœ… βœ… βœ… ❌
Redaction caching (cost savings) βœ… ❌ βœ… ❌
Open source (MIT) βœ… ❌ ❌ βœ…
Price Free OSS $$$ /mo $$$ /mo Free

When to Use Tokenectomy:

  • βœ… You use AI coding agents (Claude, Cursor, Cline, etc.)
  • βœ… You care about privacy & local-first processing
  • βœ… You want to reduce LLM token costs
  • βœ… You're worried about secret leakage in logs

When to Use Something Else:

  • ❌ You only need static secret scanning β†’ use truffleHog, detect-secrets
  • ❌ You need real-time monitoring dashboards β†’ use Datadog, New Relic, Splunk
  • ❌ Your error logs are naturally <100 tokens β†’ overhead not worth it
  • ❌ You're fully air-gapped β†’ Actually Tokenectomy is perfect! (100% local processing)

Frequently Asked Questions

Q: Does Tokenectomy send my logs to external servers?

A: No. All redaction, parsing, and filtering happens locally on your machine. The only network call is to your chosen LLM (OpenAI, Anthropic, Ollama) after sanitization is complete. See SECURITY.md for the zero-knowledge guarantee.


Q: What secrets does Tokenectomy redact?

A: GitHub PATs, AWS keys, OpenAI/Anthropic API keys, JWTs, database connection strings (PostgreSQL, MySQL, MongoDB, Redis), private SSH keys, Slack/Discord webhooks, and more. Full list in src/redact.rs.


Q: What if my secret doesn't match the redaction patterns?

A: File an issue with an example (sanitized). We'll add the pattern. For now, you can add custom patterns in ~/.tokenectomy.toml (feature coming in v1.3).


Q: Is Tokenectomy safe for production?

A: Yes. Written in safe Rust (zero unsafe code in security paths), ReDoS-immune, and audited via RustSec. See SECURITY.md for full details.


Q: Can I use Tokenectomy offline?

A: Yesβ€”except Stack Overflow search. Use --local-only flag to disable all network access (except your LLM).


Q: How do I remove Tokenectomy?

A: Simply uninstall:

npm uninstall -g tokenectomy-razor
# OR
cargo uninstall tokenectomy

Zero config cleanup neededβ€”no files left behind.


Q: Can I use Tokenectomy in my CI/CD pipeline?

A: Yes! Use the GitHub Marketplace action (see Quick Start section) or the Docker container. Works with GitHub Actions, GitLab CI, Jenkins, etc.


Q: What's the difference between Razor (OSS) and Sentinel (Commercial)?

A: Razor is the free, community version with all essential features. Sentinel adds advanced capabilities like tree-sitter AST healing, anti-hallucination guards, and time-machine undo. See Edition Comparison below.


Edition Comparison

Capability Razor (Community OSS) Sentinel (Commercial Tier)
Framework Log Filtering Yes Yes
Polyglot Trace Extraction (4 Languages) Yes Yes (7 Languages)
AI Reverse Proxy Gateway (--proxy) Yes Yes
Stack Overflow Integration Yes Yes
SHA-256 Idempotency Cache Yes Yes
ReDoS-Safe Secret Redaction Yes Yes
MCP Protocol Server (JSON-RPC) Yes Yes
Bundled Agent Skills 2 Skills (Spec TDD & Fuzzer) Full 4 Skills Suite
Tree-sitter AST Syntax Healing No Yes
Anti-Hallucination Scope Guard No Yes
Automated Test Rollback Loop No Yes
Multi-File Atomic Transactions No Yes
Time Machine Undo Engine (--undo) No Yes
True Ectomy Deep Surgery Engine No Yes
Live DB Port & Docker Diagnostics No Yes

Interested in Sentinel? View pricing & features


Roadmap

Feature Status Target Version
Java/Kotlin extractor πŸ”„ In Progress v1.2
Go extractor improvements πŸ”„ In Progress v1.2
Custom redaction rules (TOML config) πŸ“‹ Planned v1.3
VS Code extension πŸ“‹ Planned v1.4
Tree-sitter AST healing βœ… Sentinel (Paid) Now
Multi-file atomic transactions βœ… Sentinel (Paid) Now
Time machine undo engine βœ… Sentinel (Paid) Now

Security & Reliability Invariants

  • Zero-Knowledge Processing: All scanning and redaction occurs on local hardware before data leaves the system boundary.
  • ReDoS Immunity: All pattern matchers utilize finite automaton evaluation with linear time guarantees. Verified in benchmarks.
  • Path Traversal Isolation: File operations are strictly locked within workspace boundaries via WorkspaceBoundary security module.
  • Memory Safety: Implemented in safe Rust with bounded stream readers (.take()) preventing resource exhaustion attacks.
  • Audit Verification: Continuous dependency auditing maintained via RustSec advisory databases.

Vulnerability Disclosure: See SECURITY.md for responsible disclosure procedures.


Contributing

Found a bug? Have a feature request? Want to add support for a new language?

  1. Issues: github.com/Tokenectomy-Labs/Tokenectomy/issues
  2. Pull Requests: Fork, create a feature branch, and submit a PR with tests
  3. Security: See SECURITY.md for private vulnerability disclosure

See CONTRIBUTING.md for detailed contribution guidelines.


Resources

  • πŸ“– Documentation β€” Full guides, API reference, and integration tutorials
  • πŸ“‹ Changelog β€” Release history and notable changes
  • 🀝 Contributing β€” How to contribute, development workflow, and testing
  • πŸ”’ Security Policy β€” Vulnerability disclosure and audit details
  • πŸ—οΈ Architecture β€” Internal design and system architecture

License

MIT License. See LICENSE for full terms.


Made with ❀️ by @daffa2555

Questions? Open an issue or start a discussion on GitHub.