testlint 0.1.0

A comprehensive toolkit for profiling and coverage reporting across multiple programming languages
Documentation

Testlint SDK

A comprehensive toolkit for profiling and coverage reporting across multiple programming languages.

Supported Languages

  • Python (.py) - Python 3.10+ recommended (3.6+ minimum), py-spy for runtime profiling
  • Go (.go) - Requires Go 1.16+ for runtime profiling
  • TypeScript/JavaScript (.ts, .js) - Requires Node.js v12.0.0+ for runtime profiling
  • Java/JVM (.java, .jar, .class) - Java 11+ recommended (Java 8+ with commercial features), uses JFR
  • Kotlin (.kt) - Uses Java/JVM tooling (JFR, JaCoCo)
  • Scala (.scala) - Uses Java/JVM tooling (JFR, JaCoCo)

Features

🔍 Profiler

Continuous runtime profiling for 9 languages: Python, Go, TypeScript/JavaScript, Java, Rust, C#, Ruby, PHP, and C++.

Supported Languages:

  • Python (py-spy) - ✅ Flamegraphs, PID attach
  • Go (pprof) - ✅ CPU profiling, PID attach via HTTP endpoint
  • TypeScript/JavaScript (Node --cpu-prof) - ✅ V8 profiling, PID attach via V8 Inspector
  • Java (JFR) - ✅ Flight Recorder, PID attach via jcmd
  • Rust (cargo-flamegraph/perf) - ✅ Flamegraphs, Linux PID attach
  • C# (dotnet-trace) - ✅ Trace files, PID attach
  • Ruby (rbspy) - ✅ Flamegraphs, PID attach
  • PHP (Xdebug) - ✅ Cachegrind files
  • C++ (perf/valgrind) - ✅ perf.data, callgrind, Linux PID attach

Profiling Outputs:

  • Flamegraphs (SVG) - Interactive CPU visualization
  • Trace files (.nettrace, .jfr) - For PerfView, speedscope
  • Cachegrind files - For KCacheGrind/QCacheGrind
  • perf.data - For Linux perf tools
  • CPU profiles (.cpuprofile) - For Chrome DevTools

🎯 Coverage Orchestrator

Automatically wrap code execution with coverage collection - run your tests, collect coverage, and optionally upload - all in one command!

Features:

  • ✅ Auto-instrumentation - Automatically wraps code with coverage tools
  • ✅ Multi-format output - JSON, XML, LCOV, HTML
  • ✅ Integrated upload - Optionally upload coverage after collection
  • ✅ CI-friendly - Works in any CI/CD environment

Currently Supported:

  • Python (coverage.py) - ✅ Ready
  • JavaScript/TypeScript (Istanbul/NYC) - ✅ Ready
  • Java (JaCoCo) - ✅ Ready
  • Go (go test + go build -cover for applications) - ✅ Ready
  • Rust (cargo-llvm-cov) - ✅ Ready
  • C# (dotnet test + coverlet) - ✅ Ready
  • Ruby (SimpleCov) - ✅ Ready
  • PHP (PHPUnit + Xdebug/PCOV) - ✅ Ready
  • C++ (gcov/lcov or llvm-cov) - ✅ Ready

🌐 Browser Coverage (NEW!)

Collect JavaScript coverage from browser testing!

Generates wrapper scripts for Playwright and Puppeteer to automatically collect JavaScript coverage during E2E and integration tests in real browsers.

Supported Tools:

  • Playwright - ✅ Full support with automatic coverage collection
  • Puppeteer - ✅ Full support with CoverageCollector class

Features:

  • Automatic V8 coverage collection during browser tests
  • Converts V8 format to Istanbul/NYC format
  • Works with any browser (Chrome, Firefox, WebKit via Playwright)
  • Minimal test code changes required
  • Integrates with existing test suites

Quick Start:

# Generate Playwright coverage wrapper
./target/release/browser_coverage playwright --example

# Generate Puppeteer coverage wrapper
./target/release/browser_coverage puppeteer --example

# Run tests - coverage collected automatically
npx playwright test

# View HTML report
npx nyc report --reporter=html

Use Cases:

  • E2E test coverage for SPAs (React, Vue, Angular)
  • Integration test coverage in staging environments
  • QA manual testing with coverage tracking
  • Client-side JavaScript coverage for full-stack apps

⚡ Runtime Coverage (NEW!)

Attach to running processes to collect code coverage in real-time!

Unlike traditional coverage (which wraps new test runs), runtime coverage attaches to already-running processes to measure what code is being executed in production or long-running applications.

Supported Languages:

Full Runtime Injection (works on any running process):

  • Python - Injects coverage.py into running process (requires pyrasite or gdb)
  • Java - Attaches JaCoCo agent to running JVM
  • C# - Uses dotnet-coverage to attach to .NET processes
  • Ruby - Injects SimpleCov into running process (requires debase or gdb)
  • JavaScript/TypeScript - Uses V8 Inspector Protocol (requires --inspect flag)

Partial Support (requires pre-instrumented builds):

  • Go - ⚠️ Requires process built with -cover flag
  • Rust - ⚠️ Requires LLVM coverage instrumentation (-C instrument-coverage)
  • PHP - ⚠️ Requires Xdebug/PCOV loaded at startup (may need restart)
  • C++ - ⚠️ Requires gcov compilation (-fprofile-arcs -ftest-coverage)

Use Cases:

  • Production coverage monitoring
  • Long-running server coverage
  • Microservice coverage collection
  • Integration test coverage without restart

Examples:

# Full runtime injection (works on any process)
./target/release/runtime_coverage python --pid 12345
./target/release/runtime_coverage java --pid 5678
./target/release/runtime_coverage csharp --pid 9999
./target/release/runtime_coverage ruby --pid 3333
./target/release/runtime_coverage javascript --pid 4444

# Partial support (requires pre-instrumented build)
./target/release/runtime_coverage go --pid 5555      # Process built with -cover
./target/release/runtime_coverage rust --pid 6666    # Built with -C instrument-coverage
./target/release/runtime_coverage php --pid 7777     # Xdebug enabled
./target/release/runtime_coverage cpp --pid 8888     # Built with gcov flags

Important Notes:

  • Go/Rust/C++: Binary must be compiled with coverage flags from the start
  • PHP: Xdebug must be enabled before process starts (cannot inject mid-execution)
  • For processes not instrumented: Use Coverage Orchestrator instead to wrap execution

📊 Coverage Upload Tool

Upload existing coverage reports from various testing frameworks to your backend.

Supported Coverage Formats:

  • Python: coverage.py (JSON/LCOV)
  • Java: JaCoCo, Cobertura (XML)
  • JavaScript/TypeScript: Jest, Istanbul, NYC (LCOV)
  • Go: go test coverage (coverage.out)

Backend Integration

  • 📊 Automatic profile reporting to backend API
  • ⏱️ Configurable intervals (default: 5 minutes)
  • 🔐 API key authentication support
  • 💾 Protobuf & JSON formats (Protobuf is 70% smaller)
  • 🌐 Default endpoint: https://quality-web-app.pages.dev/api/v0/profiles
  • 📝 See REPORTING.md for detailed documentation

Installation

# Clone repository
git clone https://github.com/yourusername/testlint-sdk.git
cd testlint-sdk

# Build all binaries
cargo build --release

# Binaries will be in target/release/
# - profiler: Runtime profiling tool (CPU/performance profiling)
# - runtime_coverage: Runtime coverage collector (attach to running processes)
# - run_with_coverage: Coverage orchestrator (wrap test execution)
# - upload_coverage: Coverage upload tool
# - browser_coverage: Browser testing coverage setup (Playwright/Puppeteer)

Quick Start

Coverage Upload

# 1. Generate coverage with your testing tool
coverage run -m pytest
coverage json -o coverage.json

# 2. Upload to backend
./target/release/upload_coverage \
  --file coverage.json python coverage.py \
  --project my-project

# Full documentation: See COVERAGE_UPLOAD.md

Runtime Profiling

# Profile a Python application (continuous until Ctrl+C)
./target/release/profiler python server.py

# Profile with JSON export
./target/release/profiler python app.py --json profile.json

# Profile with backend reporting
export PROFILER_ENDPOINT="https://api.example.com/profiles"
export PROFILER_API_KEY="your-api-key"
./target/release/profiler python app.py

Usage

Coverage Upload Tool

# Python coverage.py
./target/release/upload_coverage \
  --file coverage.json python coverage.py \
  --project myapp

# Java JaCoCo
./target/release/upload_coverage \
  --file target/site/jacoco/jacoco.xml java jacoco \
  --project my-java-app

# JavaScript Jest/Istanbul
./target/release/upload_coverage \
  --file coverage/lcov.info javascript lcov \
  --project my-web-app

# Go
./target/release/upload_coverage \
  --file coverage.out go golang \
  --project my-go-service

# Multiple files (monorepo)
./target/release/upload_coverage \
  --file backend/coverage.json python coverage.py \
  --file frontend/coverage/lcov.info javascript lcov \
  --project fullstack-app

📖 Complete documentation: COVERAGE_UPLOAD.md

Profiler Examples

# Python - generates flamegraph, supports PID attach
./target/release/profiler python server.py
./target/release/profiler python --pid 12345

# Go - CPU profiling with pprof
./target/release/profiler go main.go

# TypeScript/JavaScript - V8 CPU profiling
./target/release/profiler typescript app.ts
./target/release/profiler javascript server.js

# Java - Java Flight Recorder
./target/release/profiler java MyApp.java
./target/release/profiler java --pid 5678

# Kotlin - Uses Java/JVM tooling (JFR)
./target/release/profiler kotlin MyApp.kt
./target/release/profiler kotlin --pid 5678

# Scala - Uses Java/JVM tooling (JFR)
./target/release/profiler scala MyApp.scala
./target/release/profiler scala --pid 5678

# Rust - flamegraph via cargo-flamegraph or perf
./target/release/profiler rust src/main.rs
./target/release/profiler rust --pid 9999  # Linux only

# C# - dotnet-trace profiling
./target/release/profiler csharp Program.cs
./target/release/profiler csharp --pid 11111

# Ruby - rbspy flamegraph
./target/release/profiler ruby server.rb
./target/release/profiler ruby --pid 22222

# PHP - Xdebug cachegrind files
./target/release/profiler php index.php

# C++ - perf or valgrind/callgrind
./target/release/profiler cpp ./myapp
./target/release/profiler cpp --pid 33333  # Linux only

# With backend reporting (sends profiles every 5 minutes)
./target/release/profiler python server.py \
  --report-endpoint https://quality-web-app.pages.dev/api/v0/profiles \
  --report-interval 60 \
  --api-key your-api-key

# Using environment variables
export PROFILER_ENDPOINT=https://quality-web-app.pages.dev/api/v0/profiles
export PROFILER_API_KEY=your-api-key
./target/release/profiler python server.py

Backend Reporting

The profiler can automatically send profiling data to a backend API for centralized monitoring and analysis.

Quick Start:

# Enable reporting with CLI options
cargo run -- python app.py --report-endpoint https://quality-web-app.pages.dev/api/v0/profiles

# Or use environment variables
export PROFILER_ENDPOINT=https://quality-web-app.pages.dev/api/v0/profiles
export PROFILER_API_KEY=your-secret-key
cargo run -- python app.py

Configuration Options:

  • --report-endpoint <url> - Backend API URL (default: https://quality-web-app.pages.dev/api/v0/profiles)
  • --report-interval <secs> - Report interval in seconds (default: 300)
  • --api-key <key> - API key for authentication

Environment Variables:

  • PROFILER_ENDPOINT - Backend endpoint URL
  • PROFILER_INTERVAL - Report interval
  • PROFILER_API_KEY - API key

📖 See REPORTING.md for complete documentation, examples, and backend implementation guide.

JVM Languages (Java, Kotlin, Scala)

All JVM languages use the same underlying tooling since they compile to JVM bytecode:

Profiling: Uses Java Flight Recorder (JFR)

  • Works with any JVM language (Java, Kotlin, Scala, Groovy, Clojure)
  • Supports PID attach via jcmd
  • Requires Java 11+ (free) or Java 8+ with commercial features enabled

Runtime Coverage: Uses JaCoCo agent

  • Attaches to running JVM processes
  • Collects bytecode-level coverage
  • Works for all JVM languages

Coverage Orchestrator: Uses JaCoCo

  • Instruments tests at runtime
  • Generates XML/HTML reports

Examples:

# Kotlin profiling
./target/release/profiler kotlin MyApp.kt
./target/release/profiler kotlin --pid 12345

# Scala profiling
./target/release/profiler scala MyApp.scala
./target/release/profiler scala --pid 12345

# Kotlin runtime coverage
./target/release/runtime_coverage kotlin --pid 12345

# Kotlin coverage orchestration
./target/release/run_with_coverage kotlin gradle test

Python Runtime Profiling

For Python files, runtime profiling uses py-spy, a sampling profiler that can attach to running processes.

Requirements:

  • Python 3.10+ (recommended) or Python 3.6+ (minimum)
  • py-spy 0.3.0 or later
  • May require sudo on macOS/Linux for process attachment

Installation:

# Via pip
pip install py-spy

# Via cargo
cargo install py-spy

# Via conda
conda install -c conda-forge py-spy

Usage:

# Profile a Python script for 10 seconds
cargo run -- python script.py --runtime 10

# Profile server simulation
cargo run -- python examples/server_simulation.py --runtime 5

# With JSON export
cargo run -- python script.py --runtime 10 --json output.json

How it works:

  • Launches Python script as subprocess
  • Attaches py-spy to collect sampling data
  • Shows hot functions and execution counts
  • Display coverage-like metrics showing CPU time per function
  • Works with native extensions (C, Rust)

Version Check:

# Check Python version
python3 --version
# Should show 3.10.0 or later (recommended)

# Check py-spy installation
py-spy --version
# Should show 0.3.0 or later

Supported Python Versions:

  • ✅ Python 3.13 (Latest - October 2024)
  • ✅ Python 3.12 (Stable - October 2023)
  • ✅ Python 3.11 (Stable, Fast - October 2022)
  • ✅ Python 3.10 (LTS until 2026 - October 2021)
  • ✅ Python 3.9 (LTS until 2025 - October 2020)
  • ⚠️ Python 3.8 (EOL October 2024 - works but upgrade recommended)
  • ⚠️ Python 3.7 and earlier (EOL - works but not recommended)

For detailed version information and troubleshooting, see PYTHON_VERSION_REQUIREMENTS.md.

TypeScript/JavaScript Runtime Profiling

For TypeScript and JavaScript files, profiling uses Node.js's built-in CPU profiler:

Requirements:

  • Node.js v12.0.0 or later (v12.0.0 introduced --cpu-prof)
  • No external tools needed

Usage:

# Profile a JavaScript file
cargo run -- javascript script.js

# Profile a TypeScript file (will run as-is with Node.js)
cargo run -- typescript app.ts

# Attach to a running Node.js process (NEW!)
cargo run -- javascript --pid 12345

# Multiple language aliases supported
cargo run -- node server.js
cargo run -- js app.js
cargo run -- ts index.ts

PID Attach Profiling:

NEW! TypeScript/JavaScript now supports PID attach via V8 Inspector Protocol:

  • Automatically detects or enables inspector (port 9229-9233)
  • Sends SIGUSR1 to enable inspector if not already enabled
  • Profiles for 30 seconds and collects CPU profile
  • Works with any running Node.js process

Requirements for PID Attach:

  • Process must respond to SIGUSR1 (enabled by default on Node.js v8+)
  • Or start with --inspect flag for guaranteed inspector availability

How it works:

  • Runs your script with node --cpu-prof
  • Collects V8 CPU profile data automatically
  • Parses the generated .cpuprofile JSON file
  • Shows hot functions with file locations and line numbers
  • Cleans up temporary profile files

Version Check:

# Check if your Node.js version supports profiling
node --version
# Should show v12.0.0 or later

# Test that --cpu-prof flag is available
node --help | grep cpu-prof

For detailed version information and troubleshooting, see NODEJS_VERSION_REQUIREMENTS.md.

Export to Common JSON Format

Export profiling data to a standardized JSON format for integration with other tools:

# Static analysis only
cargo run -- python examples/sample.py --json output.json

# With runtime profiling
cargo run -- python examples/sample.py --runtime 5 --json output.json

The common format includes:

  • Static metrics (LOC, functions, classes, complexity)
  • Runtime metrics (execution counts, hot functions, sampling data)
  • Timestamps and metadata
  • Language-agnostic structure for cross-language analysis

Architecture

The project uses a modular architecture with:

  • Core Profiler Module (src/profiler/mod.rs): Defines the LanguageProfiler trait and ProfileResult structure
  • Language-Specific Profilers:
    • src/profiler/python.rs: Python code analysis
    • src/profiler/go.rs: Go code analysis
    • src/profiler/typescript.rs: TypeScript code analysis

Each language profiler implements the LanguageProfiler trait and uses regex patterns to detect language-specific constructs.

Extending

To add support for a new language:

  1. Create a new profiler module (e.g., src/profiler/rust.rs)
  2. Implement the LanguageProfiler trait
  3. Add the language to the Language enum in src/profiler/mod.rs
  4. Update the match statement in Profiler::profile()
  5. Update the CLI argument parsing in src/main.rs

Dependencies

  • regex: For pattern matching in source code analysis
  • serde & serde_json: For parsing py-spy profiling output

Requirements

Core

  • Rust 1.70 or later
  • Cargo

For Python Runtime Profiling (Optional)

  • Python 3.10 or later (Recommended, 3.6+ minimum)
  • py-spy 0.3.0+: Install with pip install py-spy
  • May require sudo on macOS/Linux for attaching to processes
  • See PYTHON_VERSION_REQUIREMENTS.md for detailed version information

For Go Runtime Profiling (Optional)

  • Go 1.16 or later
  • Go toolchain must be in PATH

For TypeScript/JavaScript Runtime Profiling (Optional)

  • Node.js v12.0.0 or later (Released April 2019)
  • The --cpu-prof flag was introduced in v12.0.0
  • Recommended: Node.js v18 LTS or v20 LTS
  • See NODEJS_VERSION_REQUIREMENTS.md for detailed version information

External Tools

  • py-spy (optional): Required for Python runtime profiling. Install with pip install py-spy
  • Node.js v12+ (optional): Required for TypeScript/JavaScript runtime profiling