tailtriage-cli

tailtriage-cli loads tailtriage run artifacts and turns them into a triage report.

Install it after capture instrumentation is in place.

The binary name is:

tailtriage

What this tool does

tailtriage-cli owns the command-line artifact-analysis contract:

• load a captured artifact
• validate schema compatibility
• produce JSON or human-readable triage output
• invoke tailtriage-analyzer on loaded artifacts and rank likely bottleneck families
• emit evidence and next checks

The output is intended to guide the next investigation step. It does not prove root cause on its own.

Installation

cargo install tailtriage-cli

Minimal usage

Default text output:

tailtriage analyze tailtriage-run.json

Machine-readable JSON output:

tailtriage analyze tailtriage-run.json --format json

Import completed tailtriage tt.* tracing span JSONL into Run JSON:

tailtriage import tracing-spans-jsonl spans.jsonl --service checkout --output tailtriage-run.json

With optional metadata flags, strict validation, and explicit format:

tailtriage import tracing-spans-jsonl spans.jsonl --service checkout --output tailtriage-run.json --service-version v1 --run-id run-42 --strict \

  --input-format tailtriage-span-jsonl

tailtriage import tracing-spans-jsonl imports completed tailtriage tt.* tracing span JSONL into Run JSON (not Report JSON).

Accepted tt.* field types match tracing intake:

• tt.success: optional bool; strings "true" and "false" are also accepted case-insensitively.
• tt.depth_at_start: optional non-negative integer. Do not record it with debug formatting.
• tt.outcome: optional non-empty string.
• tt.kind, tt.request_id, tt.route, tt.stage, and tt.queue: scalar strings. tt.request_id must be the unique tailtriage request ID for one completed logical request/work item in one Run, not a broad external trace ID that can repeat.

Recommended stable input format is the tailtriage wrapper JSONL shape:

{"format":"tailtriage.tracing-span.v1","span":{...}}

--input-format values:

• tailtriage-span-jsonl
• compatible

Behavior:

• tailtriage-span-jsonl enforces wrapper-only parsing.
• tailtriage-span-jsonl is the default library contract: non-wrapper non-empty JSON records are hard errors.
• compatible is only for pre-stable/internal normalized completed-span shapes with explicit start/end timestamps; it is not auto-detection and not generic tracing JSON import.
• Close-event/fmt-like tracing log envelopes are unsupported import input.
• Ordinary tracing_subscriber::fmt().json() logs are unsupported and rejected; timing is not guessed from JSONL line receive time.

After import, run analysis separately:

tailtriage analyze tailtriage-run.json

Zero-request imports fail by design (the CLI loader requires at least one request).

When paths include spaces, quote them in shell usage:

tailtriage import tracing-spans-jsonl "fixtures/tracing spans.jsonl" --service checkout --output "runs/imported run.json"

Import behavior checklist:

• Imports completed tailtriage tt.* tracing span JSONL records in the documented shape.
• Writes Run JSON through the normal local JSON artifact writer, not Report JSON.
• Keeps analysis as a separate step: tailtriage analyze tailtriage-run.json.
• Prints import warnings to stderr as warning: ....
• Uses the same CaptureMode/CaptureLimits semantics as native capture for request/stage/queue evidence retention.
• Exposes request/stage/queue limit overrides because those are the evidence types offline CLI tracing import ingests.
• Does not expose runtime-snapshot or in-flight-snapshot limit flags because this import path does not ingest those evidence types.
• Does not fabricate runtime snapshots; executor/blocking-pressure interpretation remains limited unless runtime snapshots are also captured, for example via Tokio runtime sampling.
• Treats malformed JSON input as fatal.
• In non-strict mode, skips syntactically valid malformed/incomplete tt.* records with warning: ... lines.
• Prefers complete run-relative monotonic offsets when deriving or validating elapsed duration; Unix-ms bounds are the fallback when complete run-relative offsets are absent.
• Treats duration_us as authoritative elapsed-time evidence when supplied and consistent with the selected timing source.
• When duration_us is absent, derives duration from complete run-relative offsets first, then from Unix-ms wall-clock bounds.
• Rejects duration_us mismatches against the selected timing source in --strict mode.
• Warns but keeps duration_us in non-strict mode when supplied duration and the selected timing source differ beyond tolerance.
• Requires --service to be non-empty and not whitespace-only.
• Fails when zero request events would be written, such as unrelated-only input or all-skipped malformed tt.* input, because tailtriage analyze requires at least one request in CLI-loaded run artifacts.
• Applies the same non-empty-request rule before persisting completed-span JSONL artifacts in tracing intake sessions.

tailtriage analyze <run.json> --format json emits the same pretty Report JSON as tailtriage_analyzer::render_json_pretty. Add --strict-artifact when you want analysis to fail on duplicate completed request IDs or orphan stage/queue request IDs instead of using the default permissive warning behavior.

The CLI artifact loader requires at least one request event in requests. This is a CLI artifact-loading rule, not an in-process tailtriage-analyzer requirement for already-constructed Run values. CLI input is Run artifact JSON from disk. CLI does not consume Report JSON as input.

Analyzer tuning flags

Start with default analyzer behavior first.

• --analyzer-config <path> loads analyzer options from TOML ([analyzer], schema_version = 1).
• --analyzer-set PATH=VALUE applies one override (repeatable).
• --help-analyzer-options prints supported override paths and value formats.

Precedence:

1. analyzer defaults
2. options loaded from --analyzer-config
3. one or more --analyzer-set PATH=VALUE overrides (last assignment to the same path wins)

Override parsing/validation errors fail fast so misspelled paths or invalid values are rejected rather than silently ignored.

Run artifact JSON remains CLI input. Report JSON remains analyzer/CLI output. Analyzer tuning changes report interpretation, not captured artifact contents.

How to read the result

Read output in this order:

1. primary_suspect.kind
2. primary_suspect.evidence[]
3. primary_suspect.next_checks[]

Then run one targeted check, change one thing, and re-run under comparable load.

Representative output shape

{
  "request_count": 250,
  "p50_latency_us": 782227,
  "p95_latency_us": 1468239,
  "p99_latency_us": 1518551,
  "p95_queue_share_permille": 982,
  "p95_service_share_permille": 267,
  "inflight_trend": null,
  "warnings": [],
  "evidence_quality": {
    "request_count": 250,
    "queue_event_count": 250,
    "stage_event_count": 250,
    "runtime_snapshot_count": 0,
    "inflight_snapshot_count": 0,
    "requests": "present",
    "queues": "present",
    "stages": "present",
    "runtime_snapshots": "missing",
    "inflight_snapshots": "missing",
    "truncated": false,
    "dropped_requests": 0,
    "dropped_stages": 0,
    "dropped_queues": 0,
    "dropped_inflight_snapshots": 0,
    "dropped_runtime_snapshots": 0,
    "quality": "strong",
    "limitations": ["Runtime snapshots are missing, limiting executor and blocking-pressure interpretation."]
  },
  "primary_suspect": {
    "kind": "application_queue_saturation",
    "score": 90,
    "confidence": "high",
    "evidence": ["Queue wait at p95 consumes 98.2% of request time."],
    "next_checks": ["Inspect queue admission limits and producer burst patterns."],
    "confidence_notes": []
  },
  "secondary_suspects": [],
  "route_breakdowns": [],
  "temporal_segments": []
}

inflight_trend may be null when no in-flight gauges were captured.

route_breakdowns is always present in JSON output and is usually an empty array. It is populated only when at least two captured routes have enough completed requests and route-level context adds signal, such as different route-level primary suspects or a large route p95 latency spread. The global primary_suspect remains the primary full-run triage lead. Route breakdowns are supporting context only. They use route-attributed request, queue, and stage events. Runtime snapshots and in-flight gauges are global signals, so they are intentionally not attributed to individual routes. Route-level summaries do not prove per-route root cause.

temporal_segments is always present in JSON output and is usually an empty array. It is populated only when conservative within-run early/late checks detect material signal movement. The global primary_suspect remains global and unchanged by segment generation. Temporal segments are within-run hints, not proof of phase-specific root cause. Report warnings can explicitly call out large early/late p95 movement. Temporal segments prefer run-relative monotonic offsets when present. Older or imported artifacts without complete run-relative offsets fall back to Unix-ms timestamp windows. Runtime and in-flight attribution remains approximate when samples are sparse or early/late windows overlap under concurrency.

What the report contains

A report can include:

• request count
• request latency percentiles (p50, p95, p99)
• p95 queue/service share summaries
• optional in-flight trend summary
• report warnings from analysis/report generation (for example truncation-related)
• structured evidence quality coverage/status summary
• primary and secondary suspects

tailtriage analyze also prints loader/lifecycle warnings to stderr before the report. Those warnings are surfaced separately; they are not merged into the report warnings field.

Each suspect includes:

• kind
• score
• confidence
• evidence[]
• next_checks[]
• confidence_notes[] (present and empty unless evidence-aware caps affect confidence, or explicit ambiguity applies)

Artifact compatibility contract

The tailtriage analyze workflow expects a supported tailtriage run artifact with minimum required content.

Current contract:

• top-level schema_version is required
• missing schema_version is rejected
• non-integer schema_version is rejected
• unsupported schema_version is rejected
• current supported schema version is 1
• requests must contain at least one request event
• artifacts with an empty requests array are rejected by the CLI loader

For Rust in-process usage, use tailtriage-analyzer directly (analyze_run, render_text, typed Report). The stricter non-empty requests rule applies to CLI artifact loading from disk. Loader, parse, validation, and render errors return a non-zero process exit through the CLI.

Important interpretation notes

• suspects are investigation leads, not proof of root cause
• truncation warnings mean the diagnosis is based on partial retained data
• unfinished lifecycle warnings printed by the CLI indicate some requests were not completed cleanly
• p95_queue_share_permille and p95_service_share_permille are independent percentile summaries and do not need to sum to 1000

Scoring and warning behavior

Suspect ranking uses deterministic, proportional, evidence-aware scoring (0-100), not fixed suspect priority.

• Scores rank suspects inside one report; they are not probabilities.
• Confidence is score-derived ranking strength and may be evidence-quality capped; it is not causal certainty.
• confidence_notes[] explain caps, including sparse samples, truncation, missing instrumentation, ambiguous top scores, and partial-vs-missing runtime snapshot limits.
• Strong downstream tail-stage contribution can outrank weak blocking/runtime signals.
• Strong queue pressure remains a high-confidence lead when queue share/depth evidence is dominant.

How to read before/after runs:

• Compare p95 latency movement first.
• Confirm primary suspect kind/rank and evidence direction.
• Use score movement as supporting context, not a standalone pass/fail rule.

Why a score can stay flat or rise after mitigation:

• Scores are relative to the evidence mix in each capture.
• If total latency drops but the remaining tail is still dominated by one suspect family, that suspect score can remain high or increase.
• This does not by itself mean mitigation failed when p95 and relevant evidence improve.

warnings[] may include:

• evidence-quality warnings (for example low request counts or missing signal families)
• ambiguity warnings when top suspects are genuinely close after calibration
• additive truncation warnings when capture limits drop events

Suspect kinds

The current report surface includes these suspect kinds:

• application_queue_saturation
• blocking_pool_pressure
• executor_pressure_suspected
• downstream_stage_dominates
• insufficient_evidence

When the result is insufficient_evidence

Usually the next step is to add more structure to capture:

• add queue wrappers around suspected waits
• add stage wrappers around suspected downstream work
• optionally add runtime sampling if runtime pressure is unclear
• re-run under comparable load

What this tool does not do

tailtriage-cli does not capture instrumentation data.

Use capture-side crates for that:

• tailtriage: recommended capture-side entry point
• tailtriage-core: direct instrumentation primitives
• tailtriage-controller: repeated bounded windows
• tailtriage-tokio: runtime-pressure sampling
• tailtriage-axum: Axum request-boundary integration

Persisted Run JSON intended for tailtriage analyze must include at least one completed request event; in-process library snapshots may still be zero-request for inspection.