spydecy 0.3.0

Self-hosted Python/C-to-Rust compiler-debugger with introspective debugging
Documentation

Spydecy - Self-Hosted Python/C-to-Rust Compiler-Debugger

Crates.io CI Status License: MIT OR Apache-2.0 Downloads

🎉 MVP-READY • Phase 2 Complete (80%) • Zero Overhead Proven • 28 Tests Passing (100%)

Self-hosted Python/C-to-Rust transpiler with zero FFI overhead, zero unsafe code, and user-friendly error messages. Generate idiomatic, safe Rust from Python/C extensions.

Production-Ready Features:

  • ✅ 11 Patterns: List & Dict operations (len, append, get, reverse, clear, pop, etc.)
  • ✅ 0-6% Overhead: Identical performance to hand-written Rust (benchmarked)
  • ✅ Safe Rust: Zero unsafe code, no FFI boundaries
  • ✅ Real Names: Variable names preserved from source (my_list.len() not x.len())
  • ✅ Helpful Errors: Pattern suggestions and documentation links
  • ✅ Validated: 8 real-world scenarios proven end-to-end

📦 Published Crates

Crate Version Description
spydecy 0.2.0 Main CLI application
spydecy-hir 0.2.0 Unified HIR (High-level IR)
spydecy-python 0.2.0 Python AST parser (PyO3)
spydecy-debugger 0.2.0 Introspective debugger

🎯 Quick Start

Installation

From crates.io (Recommended)

# Install the latest release
cargo install spydecy

# Verify installation
spydecy --version

For Development

Prerequisites:

  • Rust 1.75.0+
  • Python 3.10-dev: sudo apt-get install python3.10-dev
  • libclang-14-dev: sudo apt-get install libclang-14-dev
  • PMAT: cargo install pmat
# Clone repository
git clone https://github.com/noahgift/spydecy.git
cd spydecy

# Install all development tools
make install-tools

# Setup project
make setup

# Build
make build

# Run quality gates
make quality-gate

Usage

# Compile Python + C to Rust (full pipeline)
spydecy compile --python my_file.py --c my_file.c --output my_file.rs --verbose

# Quick compile (non-verbose)
spydecy compile --python my_file.py --c my_file.c --output my_file.rs

# Visualize Python AST (NEW!)
spydecy debug --visualize example.py

# Visualize C AST with CPython API detection (NEW!)
spydecy debug --visualize example.c

# Show project info and status
spydecy info

# Get help
spydecy --help

✨ What's New - Phase 2 Complete (MVP-READY)

Phase 2.5: Real-World Validation ✅

Proven on 8 realistic scenarios:

# E-commerce: Shopping cart
def add_to_cart(shopping_cart):
    return append(shopping_cart)
# → shopping_cart.push(item)  ✅ Actual variable name!

# Analytics: User history
def reverse_user_history(user_history):
    return reverse(user_history)
# → user_history.reverse()  ✅ Domain-specific naming preserved!

# Config: Settings lookup
def get_config_value(config_map):
    return get(config_map)
# → config_map.get(&key)  ✅ Idiomatic Rust!

Phase 2.2: Performance Benchmarking ✅

Target EXCEEDED: 0-6% overhead (target was <20%)

Operation Hand-Written Spydecy Difference
HashMap::get(1000) 18.449 ns 18.699 ns +1.35% ✅
Vec::clear(1000) 118.90 ns 118.72 ns -0.15% ✅
Vec::pop(1000) 92.260 ns 91.581 ns -0.74% ✅

Result: Generated code performs identically to hand-written Rust!

Phase 2.3: Error Messages ✅

User-friendly diagnostics with pattern suggestions:

❌ Cannot match Python function 'unknown' with C function 'unknown_c'

💡 Supported patterns:
  1. len() + list_length() → Vec::len()
  2. append() + PyList_Append() → Vec::push()
  3. get() + PyDict_GetItem() → HashMap::get()

📖 For custom patterns, see: https://github.com/noahgift/spydecy

Phase 2.1: Full Argument Support ✅

Real variable names flow from Python to Rust:

  • item_list → item_list.len() (not generic x.len())
  • shopping_cart → shopping_cart.push(item)
  • config_map → config_map.get(&key)
  • ✅ Shared maintenance with decy project
  • ✅ Foundation for Phase 3 (ownership analysis)

Architecture: decy-parser → adapter → spydecy CAST → Unified HIR → Rust

📚 Documentation

Start Here: Response to Gemini AI Review ⭐

Critical Documents

  1. Sprint 0: Tracer Bullet - 2-week validation sprint
  2. Incremental Debugger Roadmap - Build debugger alongside transpiler
  3. Decy Integration Plan - Phase 2 Complete ✅
  4. Pluggable C-API Architecture - Extensible C-API analysis

Full Specification

🧪 Quality Standards

Metric Target Current Enforcement
Test Coverage ≥80% 99/99 ✅ PMAT + CI
Mutation Score ≥90% TBD cargo-mutants
Complexity ≤10 CCN ✅ PMAT pre-commit
SATD Comments 0 0 ✅ PMAT (zero tolerance)
Clippy Warnings 0 CI/CD
Unsafe Code <5 per 1000 LOC Static analysis

🚀 Development

Quality Gates

# Fast quality check
make quality-fast

# Full quality gate (with coverage & mutation)
make quality-gate

# Pre-commit checks
make pre-commit

# Continuous improvement analysis
make kaizen

Testing

# Run all tests
make test

# Property-based tests (1000 cases/property)
make test-property

# Mutation testing
make mutants

# Code coverage
make coverage

Development Mode

# Auto-reload on changes
make dev

# Watch and run tests
make watch

🏗️ Project Status

Current Version: v0.2.0 (Released 2025-10-22) Current Phase: Sprint 3 Complete ✅ - Planning Sprint 4

Completed Milestones ✅

  • ✅ Sprint 0: Tracer Bullet Validation - Core assumption proven: len() unification works!
  • ✅ Sprint 2: Python Transpiler - Full Python AST parsing with PyO3 (36/36 tests)
  • ✅ v0.1.0 Release: Published to crates.io with initial foundation
  • ✅ Sprint 3: C Transpiler Foundation (51/51 tests)
    • C parser complete using clang-sys
    • CPython API pattern recognition working
    • CORE INNOVATION PROVEN: Python + C → Rust unification end-to-end!
  • ✅ v0.2.0 Release: Unification Milestone 🎉
    • Python len(x) + C list_length() → Rust Vec::len() working
    • Complete pipeline validated with production parsers
    • Boundary elimination demonstrated

Next Steps 🚀

  • Sprint 4: Cross-Layer Optimization (Est. 2-4 weeks)
    • Add more unification patterns (append, dict.get)
    • Implement C debugger visualization
    • Begin optimizer implementation
    • Performance benchmarking

Roadmap

  • v0.3.0 (Est. 4-6 weeks): Expanded patterns + optimizer
  • v0.4.0 (Est. 8-10 weeks): Full code generation pipeline
  • v1.0.0 (Est. 12-16 weeks): Production ready - Self-hosting capability

See CHANGELOG.md for detailed release notes.

🔬 Architecture

Multi-Layer Pipeline

Python Source → Python HIR ─┐
                            ├─→ Unified HIR → Optimizer → Rust
C Source → C HIR ───────────┘

Key Innovations

  1. Unified Python/C Transpilation - Leverages CPython's C implementation
  2. Introspective Debugging - Step through transpilation process
  3. Pluggable C-API - CPython, NumPy, SciPy, community plugins
  4. Self-Hosting - Compiler transpiles itself for validation

📊 Makefile Targets

make help              # Show all available targets
make install-tools     # Install all dev tools (PMAT, etc.)
make quality-gate      # Run full quality gate suite
make kaizen            # Continuous improvement analysis
make ci                # Full CI pipeline

🤝 Contributing

Development Methodology: EXTREME TDD

  1. RED: Write failing tests first
  2. GREEN: Minimal implementation
  3. REFACTOR: Meet quality gates (≤10 CCN, 0 SATD, 80%+ coverage)

Quality Requirements

  • ✅ All tests pass
  • ✅ Coverage ≥80%
  • ✅ Mutation score ≥90%
  • ✅ Complexity ≤10 CCN
  • ✅ Zero SATD comments (TODO/FIXME/HACK)
  • ✅ Zero Clippy warnings
  • ✅ Code formatted with rustfmt

Pre-Commit Hooks

Pre-commit hooks automatically enforce quality gates:

  • Code formatting
  • Clippy lints
  • PMAT complexity & SATD checks
  • Fast test suite

📜 License

Licensed under either of:

at your option.

🤝 Sister Projects

Spydecy is part of a family of transpiler projects:

  • decy: C-to-Rust transpiler with ownership inference
    • ✅ Already integrated: decy uses spydecy-debugger for C AST visualization
    • 🔄 Integration planned: See DECY-INTEGRATION-PLAN.md
    • Opportunity: Use decy's advanced C parser and ownership analysis in Spydecy

🙏 Acknowledgments

Inspired by:

  • depyler: Python-to-Rust transpiler
  • decy: C-to-Rust transpiler with extreme quality
  • bashrs: Formal verification and property testing
  • ruchy: Self-hosting and PMAT integration
  • Toyota Production System: Jidoka, Kaizen, Genchi Genbutsu

Built with EXTREME quality standards. Zero compromises. 🚀

Status: v0.2.0 Released to crates.io ✅ Achievement: 🎉 Core Innovation Proven End-to-End Current: Sprint 3 Complete - Planning Sprint 4 Next: v0.3.0 - Expanded patterns + optimizer

📦 Install now: cargo install spydecy 📖 Documentation: CHANGELOG.md 🔗 Crates.io: https://crates.io/crates/spydecy