perf-sentinel 0.4.2

CLI for perf-sentinel: polyglot performance anti-pattern detector
perf-sentinel-0.4.2 is not a library.

Analyzes runtime traces (SQL queries, HTTP calls) to detect N+1 queries, redundant calls and scores I/O intensity per endpoint (GreenOps).

Why perf-sentinel?

Performance anti-patterns like N+1 queries exist in any application that does I/O: monoliths and microservices alike. In distributed architectures, a single user request cascades across multiple services, each with its own I/O and nobody has visibility on the full path. Existing tools are either runtime-specific (Hypersistence Utils -> JPA only), heavy and proprietary (Datadog, New Relic) or limited to unit tests without cross-service visibility.

perf-sentinel takes a different approach: protocol-level analysis. It observes the traces your application produces (SQL queries, HTTP calls) regardless of language or ORM. It doesn't need to understand JPA, EF Core or SeaORM, it sees the queries they generate.

GreenOps: built-in carbon-aware scoring

Every finding includes an I/O Intensity Score (IIS): the number of I/O operations generated per user request for a given endpoint. Reducing unnecessary I/O (N+1 queries, redundant calls) improves response times and reduces energy consumption, these are not competing goals.

  • I/O Intensity Score = total I/O ops for an endpoint / number of invocations
  • I/O Waste Ratio = avoidable I/O ops (from findings) / total I/O ops

Aligned with the Software Carbon Intensity model (SCI v1.0 / ISO/IEC 21031:2024) from the Green Software Foundation. The co2.total field holds the SCI numerator (E × I) + M summed over analyzed traces, not the per-request intensity score. Multi-region scoring is automatic when OTel spans carry the cloud.region attribute. 30+ cloud regions have embedded hourly carbon intensity profiles, with monthly x hourly seasonal variation for FR, DE, GB and US-East. In daemon mode, energy estimation can be refined via Scaphandre RAPL (bare metal) or cloud-native CPU% + SPECpower (AWS/GCP/Azure) and grid intensity can be pulled live from the Electricity Maps API, with automatic fallback to the I/O proxy model. Users can supply their own hourly profiles via [green] hourly_profiles_file or tune the proxy coefficients from on-site measurements via perf-sentinel calibrate.

Note: CO₂ estimates are directional, not regulatory-grade. Every estimate carries a ~2× multiplicative uncertainty bracket (low = mid/2, high = mid×2) because the I/O proxy model is rough. perf-sentinel is a waste counter, not a carbon-accounting tool. Do not use it for CSRD or GHG Protocol Scope 3 reporting. See docs/LIMITATIONS.md for the full methodology.

How does it compare?

Criteria Hypersistence Optimizer Datadog APM New Relic APM Digma perf-sentinel
N+1 SQL detection ✅ JPA only ⚠️ Manual (trace view) ⚠️ Manual (trace view) ✅ (JVM) ✅ Polyglot
N+1 HTTP detection ❌ ⚠️ Manual (trace view) ⚠️ Manual (trace view) ⚠️ Partial ✅
Polyglot ❌ Java/JPA ✅ (per-language agents) ✅ (per-language agents) ⚠️ JVM + .NET ✅ Protocol-level
Cross-service ❌ ✅ ✅ ⚠️ Partial ✅ Trace ID
GreenOps / SCI ❌ ❌ ❌ ❌ ✅ Built-in
Lightweight N/A (lib) ❌ (~150 MB) ❌ (~150 MB) ❌ (~100 MB) ✅ (<10 MB RSS)
Open-source ❌ Commercial ❌ ⚠️ Limited free tier ⚠️ Freemium ✅ AGPL v3
CI/CD quality gate ⚠️ (manual assertions) ❌ ⚠️ (alerts, no native gate) ⚠️ ✅ Native

What does it report?

For each detected anti-pattern, perf-sentinel reports:

  • Type: N+1 SQL, N+1 HTTP, redundant query, slow SQL, slow HTTP, excessive fanout, chatty service, pool saturation or serialized calls. Cross-trace correlations are also surfaced in daemon mode
  • Normalized template: the query or URL with parameters replaced by placeholders (?, {id})
  • Occurrences: how many times the pattern fired within the detection window
  • Source endpoint: which application endpoint triggered it (e.g. GET /api/orders)
  • Suggestion: e.g. "batch this query", "use a batch endpoint", "consider adding an index"
  • Source location: when OTel spans carry code.function, code.filepath, code.lineno attributes, findings display the originating source file and line. SARIF reports include physicalLocations for inline GitHub/GitLab annotations
  • GreenOps impact: estimated avoidable I/O ops, I/O Intensity Score, structured co2 object (low/mid/high, SCI v1.0 operational + embodied terms), per-region breakdown when multi-region scoring is active

demo

Or drill into a single trace with the explain tree view, which annotates findings inline next to the offending spans:

explain tree view

Or browse traces, findings and span trees interactively with the inspect TUI (3-panel layout, keyboard navigation):

inspect TUI

Or rank SQL hotspots from a PostgreSQL pg_stat_statements export with pg-stat. Three rankings (by total time, by call count, by mean latency) help you spot queries that dominate the DB without being visible in your traces — a sign of instrumentation gaps:

pg-stat hotspots

Finally, tune the I/O-to-energy coefficients to your real infrastructure with calibrate, which correlates a trace file with measured energy readings (Scaphandre, cloud monitoring, etc.) and emits a TOML file loaded via [green] calibration_file:

calibrate workflow

Configuration (.perf-sentinel.toml):

config

Analysis report (the first GIF above scrolls through the full report; the four still frames below cover it page by page, with a small overlap so every finding appears fully on at least one page):

page 1: N+1 SQL, N+1 HTTP, redundant SQL

page 2: redundant HTTP, slow SQL, slow HTTP

page 3: excessive fanout, chatty service, pool saturation

page 4: serialized calls, GreenOps summary, quality gate

Explain mode (tree view of a single trace, perf-sentinel explain --trace-id <id>). Span-anchored findings (N+1, redundant, slow, fanout) are rendered inline next to the offending spans; trace-level findings (chatty service, pool saturation, serialized calls) are surfaced in a dedicated header above the tree:

explain tree view with excessive fanout annotation on the parent span

explain trace-level header with chatty service warning

Inspect mode (interactive TUI, perf-sentinel inspect). The findings panel header colors findings by severity; below are five still frames walking the demo fixture across the three severity levels plus a detail-panel view with its scroll feature:

inspect TUI, initial view: chatty service warning (yellow)

inspect TUI, detail panel active: top of the excessive fanout span tree

inspect TUI, detail panel scrolled down: bottom half of the fanout tree

inspect TUI, N+1 SQL critical (red): 10 occurrences, batch suggestion

inspect TUI, redundant HTTP info (cyan): 3 identical token validations

pg-stat mode (perf-sentinel pg-stat --input <pg_stat_statements.csv>): ranks SQL queries three ways (by total execution time, by call count, by mean latency). Cross-reference with your traces via --traces to spot queries that dominate the DB without showing up in instrumentation:

pg-stat: top hotspots by total time, calls and mean latency

Calibrate mode (perf-sentinel calibrate --traces <traces.json> --measured-energy <energy.csv>):

calibrate input: CSV with per-service power readings

calibrate run: warnings and per-service factors printed

calibrate output: generated TOML with calibration factors

In CI mode (perf-sentinel analyze --ci), the output is a structured JSON report:

{
  "analysis": {
    "duration_ms": 0,
    "events_processed": 10,
    "traces_analyzed": 1
  },
  "findings": [
    {
      "type": "n_plus_one_sql",
      "severity": "critical",
      "trace_id": "trace-demo-nplus-sql",
      "service": "order-svc",
      "source_endpoint": "POST /api/orders/42/submit",
      "pattern": {
        "template": "SELECT * FROM order_item WHERE order_id = ?",
        "occurrences": 10,
        "window_ms": 450,
        "distinct_params": 10
      },
      "suggestion": "Use WHERE ... IN (?) to batch 10 queries into one",
      "first_timestamp": "2025-07-10T14:32:01.000Z",
      "last_timestamp": "2025-07-10T14:32:01.450Z",
      "green_impact": {
        "estimated_extra_io_ops": 9,
        "io_intensity_score": 10.0,
        "io_intensity_band": "critical"
      },
      "confidence": "ci_batch"
    }
  ],
  "green_summary": {
    "total_io_ops": 10,
    "avoidable_io_ops": 9,
    "io_waste_ratio": 0.9,
    "io_waste_ratio_band": "critical",
    "top_offenders": [
      {
        "endpoint": "POST /api/orders/42/submit",
        "service": "order-svc",
        "io_intensity_score": 10.0,
        "io_intensity_band": "critical"
      }
    ],
    "co2": {
      "total":     { "low": 0.000512, "mid": 0.001024, "high": 0.002048, "model": "io_proxy_v3", "methodology": "sci_v1_numerator" },
      "avoidable": { "low": 0.000011, "mid": 0.000021, "high": 0.000043, "model": "io_proxy_v3", "methodology": "sci_v1_operational_ratio" },
      "operational_gco2": 0.000024,
      "embodied_gco2":    0.001
    },
    "regions": [
      {
        "status": "known",
        "region": "eu-west-3",
        "grid_intensity_gco2_kwh": 42.0,
        "pue": 1.135,
        "io_ops": 10,
        "co2_gco2": 0.000024,
        "intensity_source": "monthly_hourly"
      }
    ]
  },
  "quality_gate": {
    "passed": false,
    "rules": [
      { "rule": "n_plus_one_sql_critical_max", "threshold": 0.0, "actual": 1.0, "passed": false },
      { "rule": "n_plus_one_http_warning_max", "threshold": 3.0, "actual": 0.0, "passed": true },
      { "rule": "io_waste_ratio_max", "threshold": 0.1, "actual": 0.9, "passed": false }
    ]
  }
}

How to read the report

The CLI renders a (healthy / moderate / high / critical) qualifier next to I/O Intensity Score and I/O waste ratio. The same classification ships as sibling fields in the JSON report (io_intensity_band, io_waste_ratio_band), so downstream tools like SARIF converters, Grafana panels or IDE plugins can consume our heuristics or apply their own on the raw numbers.

IIS Band Anchor
< 2.0 healthy simple CRUD baseline (≤ 2 I/O per request)
2.0 – 4.9 moderate above baseline, worth watching (heuristic)
5.0 – 9.9 high N+1 detector's flag threshold (5 occurrences)
≥ 10.0 critical N+1 detector's CRITICAL severity escalation
I/O waste ratio Band Anchor
< 10% healthy
10 – 29% moderate
30 – 49% high default [thresholds] io_waste_ratio_max
≥ 50% critical majority of analyzed I/O is waste

JSON stability contract: the enum values above (healthy / moderate / high / critical) are stable across versions. The numeric thresholds behind them are versioned with the binary and may evolve. Consumers who want a version-independent classification should read the raw io_intensity_score and io_waste_ratio fields and apply their own bands.

For per-finding severity (Critical / Warning / Info on each detector type), see docs/design/04-DETECTION.md. For the full rationale behind the interpretation bands, see docs/LIMITATIONS.md.

Getting Started

Install from crates.io

cargo install sentinel-cli

Download a prebuilt binary

Binaries for Linux (amd64, arm64), macOS (arm64) and Windows (amd64) are available on the GitHub Releases page. macOS Intel users can run the arm64 binary via Rosetta 2.

# Example: Linux amd64
curl -LO https://github.com/robintra/perf-sentinel/releases/latest/download/perf-sentinel-linux-amd64
chmod +x perf-sentinel-linux-amd64
sudo mv perf-sentinel-linux-amd64 /usr/local/bin/perf-sentinel

Run with Docker

docker run --rm -p 4317:4317 -p 4318:4318 ghcr.io/robintra/perf-sentinel:latest

Quick demo

perf-sentinel demo

Batch analysis (CI)

perf-sentinel analyze --input traces.json --ci

Explain a trace

perf-sentinel explain --input traces.json --trace-id abc123

SARIF export (GitHub/GitLab code scanning)

perf-sentinel analyze --input traces.json --format sarif

Import from Jaeger or Zipkin

# Jaeger JSON export (auto-detected)
perf-sentinel analyze --input jaeger-export.json

# Zipkin JSON v2 (auto-detected)
perf-sentinel analyze --input zipkin-traces.json

pg_stat_statements analysis

# Analyze PostgreSQL pg_stat_statements export for SQL hotspots
perf-sentinel pg-stat --input pg_stat.csv

# Cross-reference with trace findings
perf-sentinel pg-stat --input pg_stat.csv --traces traces.json

# Scrape pg_stat_statements metrics from a postgres_exporter Prometheus endpoint
perf-sentinel pg-stat --prometheus http://prometheus:9090

Interactive inspection (TUI)

perf-sentinel inspect --input traces.json

Tempo trace ingestion

# Fetch and analyze a single trace from Grafana Tempo
perf-sentinel tempo --endpoint http://tempo:3200 --trace-id abc123

# Search and analyze recent traces by service name
perf-sentinel tempo --endpoint http://tempo:3200 --service order-svc --lookback 1h

Calibrate energy coefficients

# Tune I/O-to-energy coefficients from real measurements
perf-sentinel calibrate --traces traces.json --measured-energy rapl.csv --output calibration.toml

Query a running daemon

All query sub-actions default to colored terminal output. Use --format json for scripting.

# List recent findings (colored text by default)
perf-sentinel query findings
perf-sentinel query findings --service order-svc --severity critical

# Explain a trace tree with inline findings
perf-sentinel query explain --trace-id abc123

# Interactive TUI with live daemon data
perf-sentinel query inspect

# View cross-trace correlations
perf-sentinel query correlations

# Check daemon health
perf-sentinel query status

# JSON output for scripting
perf-sentinel query findings --format json
perf-sentinel query status --format json

Streaming mode (daemon)

perf-sentinel watch

Architecture

Deployment topologies

perf-sentinel supports three deployment models. Pick the one that fits your environment.

1. CI batch analysis (recommended starting point)

Analyze pre-collected trace files in your CI/CD pipeline. The process exits with code 1 if the quality gate fails.

# In your CI job:
perf-sentinel analyze --ci --input traces.json --config .perf-sentinel.toml

Create a .perf-sentinel.toml at your project root:

[thresholds]
n_plus_one_sql_critical_max = 0    # zero tolerance for N+1 SQL
io_waste_ratio_max = 0.30          # max 30% avoidable I/O

[detection]
n_plus_one_min_occurrences = 5
slow_query_threshold_ms = 500

[green]
enabled = true
default_region = "eu-west-3"                  # optional: enables gCO2eq conversion
embodied_carbon_per_request_gco2 = 0.001      # SCI v1.0 M term, default 0.001 g/req

# Optional per-service overrides for multi-region deployments
# (used when OTel cloud.region is absent from spans):
# [green.service_regions]
# "order-svc" = "us-east-1"
# "chat-svc"  = "ap-southeast-1"

Output formats: --format text (colored, default), --format json (structured), --format sarif (GitHub/GitLab code scanning).

2. Central collector (recommended for production)

An OpenTelemetry Collector receives traces from all services and forwards them to perf-sentinel. Zero code changes in your services.

app-1 --\
app-2 ---+--> OTel Collector --> perf-sentinel (watch)
app-3 --/

Ready-to-use files are provided in examples/:

# Start the collector + perf-sentinel
docker compose -f examples/docker-compose-collector.yml up -d

# Point your apps at the collector:
#   OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4317

perf-sentinel streams findings as NDJSON to stdout and exposes Prometheus metrics with Grafana Exemplars at /metrics (port 4318).

See examples/otel-collector-config.yaml for the full collector config with sampling and filtering options.

3. Sidecar (per-service diagnostics)

perf-sentinel runs alongside a single service, sharing its network namespace. Useful for isolated debugging.

docker compose -f examples/docker-compose-sidecar.yml up -d

The app sends traces to localhost:4317 (no network hop). See examples/docker-compose-sidecar.yml.

For language-specific OTLP instrumentation (Java, .NET, Rust), see docs/INTEGRATION.md. For the full configuration reference, see docs/CONFIGURATION.md. For the daemon HTTP query API (findings, explain, correlations, status), see docs/QUERY-API.md. For in-depth design documentation, see docs/design/.

Standards and data sources

perf-sentinel's carbon estimates rest on an auditable chain of public standards, reference datasets and peer-reviewed methodology. The authoritative per-reference citation list lives in crates/sentinel-core/src/score/carbon.rs (module docstring) and in crates/sentinel-core/src/score/carbon_profiles.rs (per-region source comments on every profile entry). This section is the narrative companion.

Standard / specification

Reference datasets

Academic methodology

  • Xu et al., Energy-Efficient Query Processing, VLDB 2010. Foundational DBMS per-operation energy benchmark that motivated the SELECT 0.5x / INSERT 1.5x / UPDATE 1.5x / DELETE 1.2x multipliers on the proxy model.
  • Tsirogiannis et al., Analyzing the Energy Efficiency of a Database Server, SIGMOD 2010. Companion benchmark establishing verb-level coefficients.
  • Siddik et al., DBJoules: Towards Understanding the Energy Consumption of Database Management Systems, 2023. Confirms 7-38% inter-operation variance across verbs, cross-validation for the per_operation_coefficients feature.
  • Guo et al., Energy-efficient Database Systems: A Systematic Survey, ACM Computing Surveys 2022. Overview of the field.
  • IDEAS 2025 framework: real-time energy estimation model for SQL queries, referenced as the direction of travel for future calibrate improvements.
  • Mytton, Lunden & Malmodin, Estimating electricity usage of data transmission networks, Journal of Industrial Ecology 2024. Source for the 0.04 kWh/GB default on the optional include_network_transport term; the paper's 0.03-0.06 kWh/GB range is the origin of the configurable network_energy_per_byte_kwh field.
  • Boavizta API / HotCarbon 2024: bottom-up server lifecycle embodied carbon model, referenced for the embodied_per_request_gco2 default calibration.

What this is not

perf-sentinel is a directional waste counter, not a regulatory carbon-accounting tool. Every CarbonEstimate ships with a { low, mid, high } 2× multiplicative uncertainty bracket because the I/O-to-energy proxy is approximate by construction. Do not use these values for CSRD reporting, GHG Protocol Scope 3 disclosures or any other compliance context. See docs/LIMITATIONS.md for the full methodology critique.

License

This project is licensed under the GNU Affero General Public License v3.0.