# Testlint SDK
A comprehensive toolkit for profiling and coverage reporting across multiple programming languages.
## Supported Languages
- **Python** (.py) - Python 3.10+ recommended (3.6+ minimum), py-spy for runtime profiling
- **Go** (.go) - Requires Go 1.16+ for runtime profiling
- **TypeScript/JavaScript** (.ts, .js) - Requires Node.js v12.0.0+ for runtime profiling
- **Java/JVM** (.java, .jar, .class) - Java 11+ recommended (Java 8+ with commercial features), uses JFR
- **Kotlin** (.kt) - Uses Java/JVM tooling (JFR, JaCoCo)
- **Scala** (.scala) - Uses Java/JVM tooling (JFR, JaCoCo)
## Features
### 🔍 Profiler
Continuous runtime profiling for **9 languages**: Python, Go, TypeScript/JavaScript, Java, Rust, C#, Ruby, PHP, and C++.
**Supported Languages:**
- **Python** (py-spy) - ✅ Flamegraphs, PID attach
- **Go** (pprof) - ✅ CPU profiling, PID attach via HTTP endpoint
- **TypeScript/JavaScript** (Node --cpu-prof) - ✅ V8 profiling, PID attach via V8 Inspector
- **Java** (JFR) - ✅ Flight Recorder, PID attach via jcmd
- **Rust** (cargo-flamegraph/perf) - ✅ Flamegraphs, Linux PID attach
- **C#** (dotnet-trace) - ✅ Trace files, PID attach
- **Ruby** (rbspy) - ✅ Flamegraphs, PID attach
- **PHP** (Xdebug) - ✅ Cachegrind files
- **C++** (perf/valgrind) - ✅ perf.data, callgrind, Linux PID attach
**Profiling Outputs:**
- Flamegraphs (SVG) - Interactive CPU visualization
- Trace files (.nettrace, .jfr) - For PerfView, speedscope
- Cachegrind files - For KCacheGrind/QCacheGrind
- perf.data - For Linux perf tools
- CPU profiles (.cpuprofile) - For Chrome DevTools
### 🎯 Coverage Orchestrator
Automatically wrap code execution with coverage collection - run your tests, collect coverage, and optionally upload - all in one command!
**Features:**
- ✅ **Auto-instrumentation** - Automatically wraps code with coverage tools
- ✅ **Multi-format output** - JSON, XML, LCOV, HTML
- ✅ **Integrated upload** - Optionally upload coverage after collection
- ✅ **CI-friendly** - Works in any CI/CD environment
**Currently Supported:**
- **Python** (coverage.py) - ✅ Ready
- **JavaScript/TypeScript** (Istanbul/NYC) - ✅ Ready
- **Java** (JaCoCo) - ✅ Ready
- **Go** (go test + go build -cover for applications) - ✅ Ready
- **Rust** (cargo-llvm-cov) - ✅ Ready
- **C#** (dotnet test + coverlet) - ✅ Ready
- **Ruby** (SimpleCov) - ✅ Ready
- **PHP** (PHPUnit + Xdebug/PCOV) - ✅ Ready
- **C++** (gcov/lcov or llvm-cov) - ✅ Ready
### 🌐 Browser Coverage (NEW!)
**Collect JavaScript coverage from browser testing!**
Generates wrapper scripts for Playwright and Puppeteer to automatically collect JavaScript coverage during E2E and integration tests in real browsers.
**Supported Tools:**
- **Playwright** - ✅ Full support with automatic coverage collection
- **Puppeteer** - ✅ Full support with CoverageCollector class
**Features:**
- Automatic V8 coverage collection during browser tests
- Converts V8 format to Istanbul/NYC format
- Works with any browser (Chrome, Firefox, WebKit via Playwright)
- Minimal test code changes required
- Integrates with existing test suites
**Quick Start:**
```bash
# Generate Playwright coverage wrapper
./target/release/browser_coverage playwright --example
# Generate Puppeteer coverage wrapper
./target/release/browser_coverage puppeteer --example
# Run tests - coverage collected automatically
npx playwright test
# View HTML report
npx nyc report --reporter=html
```
**Use Cases:**
- E2E test coverage for SPAs (React, Vue, Angular)
- Integration test coverage in staging environments
- QA manual testing with coverage tracking
- Client-side JavaScript coverage for full-stack apps
### ⚡ Runtime Coverage (NEW!)
**Attach to running processes to collect code coverage in real-time!**
Unlike traditional coverage (which wraps new test runs), runtime coverage attaches to **already-running** processes to measure what code is being executed in production or long-running applications.
**Supported Languages:**
**Full Runtime Injection** (works on any running process):
- **Python** - Injects coverage.py into running process (requires pyrasite or gdb)
- **Java** - Attaches JaCoCo agent to running JVM
- **C#** - Uses dotnet-coverage to attach to .NET processes
- **Ruby** - Injects SimpleCov into running process (requires debase or gdb)
- **JavaScript/TypeScript** - Uses V8 Inspector Protocol (requires --inspect flag)
**Partial Support** (requires pre-instrumented builds):
- **Go** - ⚠️ Requires process built with `-cover` flag
- **Rust** - ⚠️ Requires LLVM coverage instrumentation (`-C instrument-coverage`)
- **PHP** - ⚠️ Requires Xdebug/PCOV loaded at startup (may need restart)
- **C++** - ⚠️ Requires gcov compilation (`-fprofile-arcs -ftest-coverage`)
**Use Cases:**
- Production coverage monitoring
- Long-running server coverage
- Microservice coverage collection
- Integration test coverage without restart
**Examples:**
```bash
# Full runtime injection (works on any process)
./target/release/runtime_coverage python --pid 12345
./target/release/runtime_coverage java --pid 5678
./target/release/runtime_coverage csharp --pid 9999
./target/release/runtime_coverage ruby --pid 3333
./target/release/runtime_coverage javascript --pid 4444
# Partial support (requires pre-instrumented build)
./target/release/runtime_coverage go --pid 5555 # Process built with -cover
./target/release/runtime_coverage rust --pid 6666 # Built with -C instrument-coverage
./target/release/runtime_coverage php --pid 7777 # Xdebug enabled
./target/release/runtime_coverage cpp --pid 8888 # Built with gcov flags
```
**Important Notes:**
- **Go/Rust/C++**: Binary must be compiled with coverage flags from the start
- **PHP**: Xdebug must be enabled before process starts (cannot inject mid-execution)
- For processes not instrumented: Use **Coverage Orchestrator** instead to wrap execution
### 📊 Coverage Upload Tool
Upload existing coverage reports from various testing frameworks to your backend.
**Supported Coverage Formats:**
- **Python**: coverage.py (JSON/LCOV)
- **Java**: JaCoCo, Cobertura (XML)
- **JavaScript/TypeScript**: Jest, Istanbul, NYC (LCOV)
- **Go**: go test coverage (coverage.out)
### Backend Integration
- 📊 **Automatic profile reporting** to backend API
- ⏱️ **Configurable intervals** (default: 5 minutes)
- 🔐 **API key authentication** support
- 💾 **Protobuf & JSON formats** (Protobuf is 70% smaller)
- 🌐 **Default endpoint:** `https://quality-web-app.pages.dev/api/v0/profiles`
- 📝 **See [REPORTING.md](REPORTING.md) for detailed documentation**
## Installation
```bash
# Clone repository
git clone https://github.com/yourusername/testlint-sdk.git
cd testlint-sdk
# Build all binaries
cargo build --release
# Binaries will be in target/release/
# - profiler: Runtime profiling tool (CPU/performance profiling)
# - runtime_coverage: Runtime coverage collector (attach to running processes)
# - run_with_coverage: Coverage orchestrator (wrap test execution)
# - upload_coverage: Coverage upload tool
# - browser_coverage: Browser testing coverage setup (Playwright/Puppeteer)
```
## Quick Start
### Coverage Upload
```bash
# 1. Generate coverage with your testing tool
coverage run -m pytest
coverage json -o coverage.json
# 2. Upload to backend
./target/release/upload_coverage \
--file coverage.json python coverage.py \
--project my-project
# Full documentation: See COVERAGE_UPLOAD.md
```
### Runtime Profiling
```bash
# Profile a Python application (continuous until Ctrl+C)
./target/release/profiler python server.py
# Profile with JSON export
./target/release/profiler python app.py --json profile.json
# Profile with backend reporting
export PROFILER_ENDPOINT="https://api.example.com/profiles"
export PROFILER_API_KEY="your-api-key"
./target/release/profiler python app.py
```
## Usage
### Coverage Upload Tool
```bash
# Python coverage.py
./target/release/upload_coverage \
--file coverage.json python coverage.py \
--project myapp
# Java JaCoCo
./target/release/upload_coverage \
--file target/site/jacoco/jacoco.xml java jacoco \
--project my-java-app
# JavaScript Jest/Istanbul
./target/release/upload_coverage \
--file coverage/lcov.info javascript lcov \
--project my-web-app
# Go
./target/release/upload_coverage \
--file coverage.out go golang \
--project my-go-service
# Multiple files (monorepo)
./target/release/upload_coverage \
--file backend/coverage.json python coverage.py \
--file frontend/coverage/lcov.info javascript lcov \
--project fullstack-app
```
**📖 Complete documentation: [COVERAGE_UPLOAD.md](COVERAGE_UPLOAD.md)**
### Profiler Examples
```bash
# Python - generates flamegraph, supports PID attach
./target/release/profiler python server.py
./target/release/profiler python --pid 12345
# Go - CPU profiling with pprof
./target/release/profiler go main.go
# TypeScript/JavaScript - V8 CPU profiling
./target/release/profiler typescript app.ts
./target/release/profiler javascript server.js
# Java - Java Flight Recorder
./target/release/profiler java MyApp.java
./target/release/profiler java --pid 5678
# Kotlin - Uses Java/JVM tooling (JFR)
./target/release/profiler kotlin MyApp.kt
./target/release/profiler kotlin --pid 5678
# Scala - Uses Java/JVM tooling (JFR)
./target/release/profiler scala MyApp.scala
./target/release/profiler scala --pid 5678
# Rust - flamegraph via cargo-flamegraph or perf
./target/release/profiler rust src/main.rs
./target/release/profiler rust --pid 9999 # Linux only
# C# - dotnet-trace profiling
./target/release/profiler csharp Program.cs
./target/release/profiler csharp --pid 11111
# Ruby - rbspy flamegraph
./target/release/profiler ruby server.rb
./target/release/profiler ruby --pid 22222
# PHP - Xdebug cachegrind files
./target/release/profiler php index.php
# C++ - perf or valgrind/callgrind
./target/release/profiler cpp ./myapp
./target/release/profiler cpp --pid 33333 # Linux only
# With backend reporting (sends profiles every 5 minutes)
./target/release/profiler python server.py \
--report-endpoint https://quality-web-app.pages.dev/api/v0/profiles \
--report-interval 60 \
--api-key your-api-key
# Using environment variables
export PROFILER_ENDPOINT=https://quality-web-app.pages.dev/api/v0/profiles
export PROFILER_API_KEY=your-api-key
./target/release/profiler python server.py
```
### Backend Reporting
The profiler can automatically send profiling data to a backend API for centralized monitoring and analysis.
**Quick Start:**
```bash
# Enable reporting with CLI options
cargo run -- python app.py --report-endpoint https://quality-web-app.pages.dev/api/v0/profiles
# Or use environment variables
export PROFILER_ENDPOINT=https://quality-web-app.pages.dev/api/v0/profiles
export PROFILER_API_KEY=your-secret-key
cargo run -- python app.py
```
**Configuration Options:**
- `--report-endpoint <url>` - Backend API URL (default: `https://quality-web-app.pages.dev/api/v0/profiles`)
- `--report-interval <secs>` - Report interval in seconds (default: 300)
- `--api-key <key>` - API key for authentication
**Environment Variables:**
- `PROFILER_ENDPOINT` - Backend endpoint URL
- `PROFILER_INTERVAL` - Report interval
- `PROFILER_API_KEY` - API key
**📖 See [REPORTING.md](REPORTING.md) for complete documentation, examples, and backend implementation guide.**
### JVM Languages (Java, Kotlin, Scala)
All JVM languages use the same underlying tooling since they compile to JVM bytecode:
**Profiling:** Uses Java Flight Recorder (JFR)
- Works with any JVM language (Java, Kotlin, Scala, Groovy, Clojure)
- Supports PID attach via jcmd
- Requires Java 11+ (free) or Java 8+ with commercial features enabled
**Runtime Coverage:** Uses JaCoCo agent
- Attaches to running JVM processes
- Collects bytecode-level coverage
- Works for all JVM languages
**Coverage Orchestrator:** Uses JaCoCo
- Instruments tests at runtime
- Generates XML/HTML reports
**Examples:**
```bash
# Kotlin profiling
./target/release/profiler kotlin MyApp.kt
./target/release/profiler kotlin --pid 12345
# Scala profiling
./target/release/profiler scala MyApp.scala
./target/release/profiler scala --pid 12345
# Kotlin runtime coverage
./target/release/runtime_coverage kotlin --pid 12345
# Kotlin coverage orchestration
./target/release/run_with_coverage kotlin gradle test
```
### Python Runtime Profiling
For Python files, runtime profiling uses py-spy, a sampling profiler that can attach to running processes.
**Requirements:**
- Python 3.10+ (recommended) or Python 3.6+ (minimum)
- py-spy 0.3.0 or later
- May require `sudo` on macOS/Linux for process attachment
**Installation:**
```bash
# Via pip
pip install py-spy
# Via cargo
cargo install py-spy
# Via conda
conda install -c conda-forge py-spy
```
**Usage:**
```bash
# Profile a Python script for 10 seconds
cargo run -- python script.py --runtime 10
# Profile server simulation
cargo run -- python examples/server_simulation.py --runtime 5
# With JSON export
cargo run -- python script.py --runtime 10 --json output.json
```
**How it works:**
- Launches Python script as subprocess
- Attaches py-spy to collect sampling data
- Shows hot functions and execution counts
- Display coverage-like metrics showing CPU time per function
- Works with native extensions (C, Rust)
**Version Check:**
```bash
# Check Python version
python3 --version
# Should show 3.10.0 or later (recommended)
# Check py-spy installation
py-spy --version
# Should show 0.3.0 or later
```
**Supported Python Versions:**
- ✅ Python 3.13 (Latest - October 2024)
- ✅ Python 3.12 (Stable - October 2023)
- ✅ Python 3.11 (Stable, Fast - October 2022)
- ✅ Python 3.10 (LTS until 2026 - October 2021)
- ✅ Python 3.9 (LTS until 2025 - October 2020)
- ⚠️ Python 3.8 (EOL October 2024 - works but upgrade recommended)
- ⚠️ Python 3.7 and earlier (EOL - works but not recommended)
For detailed version information and troubleshooting, see [PYTHON_VERSION_REQUIREMENTS.md](PYTHON_VERSION_REQUIREMENTS.md).
### TypeScript/JavaScript Runtime Profiling
For TypeScript and JavaScript files, profiling uses Node.js's built-in CPU profiler:
**Requirements:**
- Node.js v12.0.0 or later (v12.0.0 introduced `--cpu-prof`)
- No external tools needed
**Usage:**
```bash
# Profile a JavaScript file
cargo run -- javascript script.js
# Profile a TypeScript file (will run as-is with Node.js)
cargo run -- typescript app.ts
# Attach to a running Node.js process (NEW!)
cargo run -- javascript --pid 12345
# Multiple language aliases supported
cargo run -- node server.js
cargo run -- js app.js
cargo run -- ts index.ts
```
**PID Attach Profiling:**
NEW! TypeScript/JavaScript now supports PID attach via V8 Inspector Protocol:
- Automatically detects or enables inspector (port 9229-9233)
- Sends SIGUSR1 to enable inspector if not already enabled
- Profiles for 30 seconds and collects CPU profile
- Works with any running Node.js process
**Requirements for PID Attach:**
- Process must respond to SIGUSR1 (enabled by default on Node.js v8+)
- Or start with `--inspect` flag for guaranteed inspector availability
**How it works:**
- Runs your script with `node --cpu-prof`
- Collects V8 CPU profile data automatically
- Parses the generated `.cpuprofile` JSON file
- Shows hot functions with file locations and line numbers
- Cleans up temporary profile files
**Version Check:**
```bash
# Check if your Node.js version supports profiling
node --version
# Should show v12.0.0 or later
# Test that --cpu-prof flag is available
For detailed version information and troubleshooting, see [NODEJS_VERSION_REQUIREMENTS.md](NODEJS_VERSION_REQUIREMENTS.md).
### Export to Common JSON Format
Export profiling data to a standardized JSON format for integration with other tools:
```bash
# Static analysis only
cargo run -- python examples/sample.py --json output.json
# With runtime profiling
cargo run -- python examples/sample.py --runtime 5 --json output.json
```
The common format includes:
- Static metrics (LOC, functions, classes, complexity)
- Runtime metrics (execution counts, hot functions, sampling data)
- Timestamps and metadata
- Language-agnostic structure for cross-language analysis
## Architecture
The project uses a modular architecture with:
- **Core Profiler Module** (`src/profiler/mod.rs`): Defines the `LanguageProfiler` trait and `ProfileResult` structure
- **Language-Specific Profilers**:
- `src/profiler/python.rs`: Python code analysis
- `src/profiler/go.rs`: Go code analysis
- `src/profiler/typescript.rs`: TypeScript code analysis
Each language profiler implements the `LanguageProfiler` trait and uses regex patterns to detect language-specific constructs.
## Extending
To add support for a new language:
1. Create a new profiler module (e.g., `src/profiler/rust.rs`)
2. Implement the `LanguageProfiler` trait
3. Add the language to the `Language` enum in `src/profiler/mod.rs`
4. Update the match statement in `Profiler::profile()`
5. Update the CLI argument parsing in `src/main.rs`
## Dependencies
- `regex`: For pattern matching in source code analysis
- `serde` & `serde_json`: For parsing py-spy profiling output
## Requirements
### Core
- Rust 1.70 or later
- Cargo
### For Python Runtime Profiling (Optional)
- **Python 3.10 or later** (Recommended, 3.6+ minimum)
- `py-spy 0.3.0+`: Install with `pip install py-spy`
- May require `sudo` on macOS/Linux for attaching to processes
- See [PYTHON_VERSION_REQUIREMENTS.md](PYTHON_VERSION_REQUIREMENTS.md) for detailed version information
### For Go Runtime Profiling (Optional)
- Go 1.16 or later
- Go toolchain must be in PATH
### For TypeScript/JavaScript Runtime Profiling (Optional)
- **Node.js v12.0.0 or later** (Released April 2019)
- The `--cpu-prof` flag was introduced in v12.0.0
- Recommended: Node.js v18 LTS or v20 LTS
- See [NODEJS_VERSION_REQUIREMENTS.md](NODEJS_VERSION_REQUIREMENTS.md) for detailed version information
## External Tools
- `py-spy` (optional): Required for Python runtime profiling. Install with `pip install py-spy`
- Node.js v12+ (optional): Required for TypeScript/JavaScript runtime profiling