Expand description
§tailtriage-tracing
tailtriage-tracing is a narrow tracing intake bridge for completed tt.* spans.
It helps existing tracing users produce standard tailtriage_core::Run artifacts by:
- writing Run JSON on shutdown, and/or
- writing retained, semantically valid completed-span JSONL on shutdown.
It is not:
- an observability backend,
- generic tracing log scraping,
- an OTel/OTLP pipeline,
- proof of root cause (output remains triage leads).
§When to use this crate
Use this path when your service already uses Rust tracing and already has stable per-work-item IDs that can be converted into unique tailtriage request IDs. New integrations without existing tracing/correlation should start with native tailtriage capture first.
This crate converts tracing-shaped request, stage, and queue evidence into standard tailtriage_core::Run artifacts for the normal tailtriage analyze workflow. It is not a tracing backend.
For one completed logical request/work item, every request, stage, and queue span must carry the same tt.request_id. That tt.request_id is the unique tailtriage request ID for the completed request within one Run, not necessarily a raw distributed trace ID. Child stage/queue evidence is correlated to retained request evidence by tt.request_id; missing, inconsistent, or duplicate IDs cause child evidence to be skipped, weakened, or reported as ambiguous.
External trace/correlation IDs may repeat across retries, fanout branches, batch items, or attempts. When they can repeat, derive a unique tailtriage tt.request_id first, for example by adding attempt, span, branch, or item information. Users remain responsible for meaningful instrumentation and request-boundary semantics.
§Feature flags
- Base crate: typed
SpanRecord,ImportOptions,ImportedRun, semantic constants, andrun_from_span_records(...). - Default (
jsonl): JSONL import APIs and stable wrapper parsing. live: enablesTracingRecorder,TailtriageLayer, andTracingIntakeSession.tokio: enablesTracingTokioSessionruntime-sampler coupling and includeslive(background sampler on by default; deterministic runs can calldisable_background_sampler()and inject snapshots manually).
CLI offline import workflows only need JSONL import support and do not require the live tracing_subscriber layer dependency.
The same APIs are also available through the default tailtriage crate when enabling its tracing, tracing-live, or tracing-tokio façade features.
For live tracing intake sessions, tailtriage-tracing enables optional dependencies behind feature flags, but applications that call tracing::... or tracing_subscriber::... directly still need explicit app dependencies:
cargo add tailtriage-tracing --features live
cargo add tracing tracing-subscriber
cargo add tokio --features macros,rt-multi-threadIf you use Tokio runtime sampler coupling via TracingTokioSession, use:
cargo add tailtriage-tracing --features tokio
cargo add tracing tracing-subscriber
cargo add tokio --features macros,rt-multi-thread§Recommended live session setup (live feature)
use tailtriage_tracing::TracingIntakeSession;
use tracing::Instrument as _;
use tracing_subscriber::prelude::*;
async fn work() {
// Your request work goes here.
}
#[tokio::main(flavor = "current_thread")]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
let session = TracingIntakeSession::builder("checkout-service")
.run_json_path("target/tailtriage-examples/checkout.run.json")
.completed_span_jsonl_path("target/tailtriage-examples/checkout.spans.jsonl")
.build()?;
tracing_subscriber::registry()
.with(session.layer())
.init(); // startup-only: global subscriber installation for this process
{
let request = tracing::info_span!(
"request",
tt.kind = "request",
tt.request_id = "req-1",
tt.route = "/checkout",
tt.outcome = "ok"
);
work().instrument(request).await;
} // the request span is closed before shutdown
let imported = session.shutdown()?;
let _ = imported;
Ok(())
}Install the tailtriage layer beside your existing tracing layers in the application’s normal process-wide subscriber setup; tailtriage augments your tracing pipeline rather than replacing it.
If your service already builds a subscriber in startup code, compose session.layer() onto that subscriber and install it once with your normal global install path (for example .init() during binary startup, or tracing::subscriber::set_global_default(...) with explicit error handling when needed).
set_default is scoped to the current thread and guard lifetime; service startup should install the tailtriage layer in the process-wide subscriber setup.
§Timing model
- Imported or live tracing evidence is converted to normal tailtriage Run timing fields.
- Live tracing samples finish wall time when the span closes.
- Newer live tracing output includes run-relative monotonic offsets for request, stage, and queue spans when those offsets are available; these offsets improve temporal grouping inside a captured run.
- Imported JSONL may omit run-relative offsets. When they are absent, the analyzer falls back to Unix-ms wall-clock timestamps for temporal grouping.
duration_usremains the authoritative elapsed-time evidence when supplied or recorded by live tracing.- Completed-span JSONL export preserves
duration_uswhen it is replay-safe under import precedence: complete run-relative monotonic offsets are used when present; otherwise exported Unix-ms bounds are used. When the selected timing source does not match the retained duration, export omitsduration_us; replay then derives elapsed time from the best available timing evidence. - Unix-ms timestamps are wall-clock anchors and may be coarser than durations.
- Ordinary tracing log timestamps are not enough for completed-span import; completed spans need explicit start/end timing and semantic tt.* fields.
§Direct Run JSON path
Use run_json_path(...) when you want to skip a separate import step and write Run JSON through the same robust writer path used by native capture sinks:
tailtriage analyze target/tailtriage-examples/checkout.run.jsonFor both TracingIntakeSession and TracingTokioSession, persisted-output shutdown
returns a zero-request error when no request is retained. When tracing intake warnings
exist (for example malformed tt.* fields), shutdown includes those warning messages in
the same error to guide setup corrections before rerunning capture.
§Completed-span JSONL path
Use completed_span_jsonl_path(...) when you want an offline import workflow:
tailtriage import tracing-spans-jsonl target/tailtriage-examples/checkout.spans.jsonl \
--service checkout-service \
--output target/tailtriage-examples/checkout.run.json
tailtriage analyze target/tailtriage-examples/checkout.run.jsonIf you configure both completed_span_jsonl_path(...) and run_json_path(...), each
configured file is written independently through its own temp/rename path. The two
outputs are not committed as one atomic transaction: if the second write fails, the
first output may already exist as a finalized artifact. For production workflows that
need one canonical shutdown artifact, prefer run_json_path(...). Completed-span JSONL
remains a replay/debug export, not trace archival.
§Stable JSONL wrapper format
Stable completed-span JSONL records use this wrapper:
{"format":"tailtriage.tracing-span.v1","span":{...}}format is a wrapper-level field (not a SpanRecord field).
The simple library import APIs (import_jsonl_reader / import_jsonl_path) default to this wrapper-only mode and return an error for any non-empty non-wrapper JSON record.
Use *_with_mode(..., JsonlParseMode::Compatible) only for pre-stable/internal normalized completed-span records with explicit unix-ms start/end timestamps.
Close-event/fmt-like tracing log envelopes are not supported import input.
Ordinary tracing log JSON (for example tracing_subscriber::fmt().json() output) is unsupported and rejected by import.
Import does not guess span timing from line receive time: provide explicit unix-ms start/end timestamps on completed spans.
§tt.* field convention
| Span kind | Required fields | Optional fields |
|---|---|---|
| request | tt.kind="request", tt.request_id, tt.route | tt.outcome (optional non-empty string; recommended common labels: ok, error, timeout, cancelled, rejected) |
| stage | tt.kind="stage", tt.request_id, tt.stage | tt.success |
| queue | tt.kind="queue", tt.request_id, tt.queue | tt.depth_at_start |
Accepted field types:
tt.success: optional bool; strings"true"and"false"are also accepted case-insensitively.tt.depth_at_start: optional non-negative integer. Do not record it with debug formatting.tt.outcome: optional non-empty string.tt.kind,tt.request_id,tt.route,tt.stage, andtt.queue: scalar strings.
Use a plain integer value for queue depth:
tt.depth_at_start = depth_u64tt.depth_at_start = ?depth may produce a debug-formatted value and be rejected.
For semantic string fields, tt.kind = "request" works, and tt.kind = %kind can work when kind displays as request. tt.kind = ?kind may record "request" with debug quoting and be rejected as unknown.
Missing request tt.outcome defaults to ok with a warning. Missing stage tt.success defaults to true with a warning.
Live tracing intake only tracks spans that are tailtriage candidates at span creation time. Declare tt.* fields when the span is created. If a value is filled later, declare the field with tracing::field::Empty and then call span.record(...); adding brand-new tt.* fields later with span.record(...) is not supported.
let span = tracing::info_span!(
"request",
tt.kind = "request",
tt.request_id = "req-1",
tt.route = "/checkout",
tt.outcome = tracing::field::Empty,
);
span.record("tt.outcome", "timeout");§Strict vs non-strict
- Strict mode: malformed/incomplete
tt.*span records fail import/session conversion. - Non-strict mode: malformed/incomplete records are warned and skipped where implemented.
- Duration consistency rule: complete run-relative offsets are preferred when present; Unix-ms bounds are the fallback when complete run-relative offsets are absent.
duration_usremains authoritative elapsed-time evidence when supplied and consistent with the selected timing source. If complete run-relative monotonic offsets are present, strict conversion requiresduration_usto matchfinished_at_run_us - started_at_run_uswithin2_000microseconds. If complete run-relative offsets are absent, strict conversion checksduration_usagainst(finished_at_unix_ms - started_at_unix_ms) * 1000. Non-strict conversion warns on mismatches but keeps the suppliedduration_us. Whenduration_usis absent, conversion derives elapsed time from complete run-relative offsets first, then falls back to Unix-ms wall-clock bounds. - Child stage/queue containment uses a fixed
2ms tolerance when checking whether child intervals fall inside retained request intervals. - That
2ms containment tolerance is not configurable in this release (no CLI/API knob).
§Retention and drop behavior
DEFAULT_MAX_OPEN_SPANSandDEFAULT_MAX_COMPLETED_CANDIDATE_SPANSare live-recorder memory caps for in-memory span tracking.max_open_spansbounds in-flight span tracking.max_completed_candidate_spansis a raw live-recorder memory cap for closed candidates before semantic conversion.- Under raw-cap pressure, request roots are preserved preferentially when possible.
- Child stage/queue evidence may be dropped or evicted under that pressure and is surfaced through warnings plus
truncation.limits_hit. - Request/stage/queue semantic retention uses
CaptureMode,CaptureLimits, andCaptureLimitsOverride. completed_span_jsonl_path(...)writes retained tailtriage semantic evidence as stable span-shaped JSONL on shutdown only when at least one completed request is retained.- Completed-span JSONL is retained-evidence replay/debug export for the same request/stage/queue evidence path through
tailtriage import; it is not a production trace archive. - It does not preserve lifecycle warnings, truncation counters, original span IDs, parent IDs, original span names, or non-
tt.*fields. - For production workflows that need the complete persisted triage artifact including warnings/truncation metadata, prefer
run_json_path(...). - Callers using JSONL-only export should inspect
session.shutdown()?.warnings()in the same process. - This completed-span JSONL is a narrow retained-evidence export, not a generic tracing log stream and not OTel/OTLP.
§Runtime-pressure limitation
Tracing intake import and native capture share the same CaptureMode/CaptureLimits semantics for request/stage/queue evidence retention. Offline completed tailtriage tracing span JSONL import does not fabricate runtime snapshots. Runtime-pressure evidence still requires runtime snapshots/Tokio sampler coupling. Runtime-sensitive tracing contract parity uses deterministic/manual runtime snapshots and requires non-empty runtime snapshots, scenario-specific runtime field evidence, and the explicit disabled-background-sampler lifecycle warning (via disable_background_sampler() + record_runtime_snapshot(...)). It does not rely on ambient sampler metadata/noise.
Persisted Run JSON intended for tailtriage analyze must include at least one completed request event; shutdown fails for persisted-output sessions when zero completed requests are retained. When intake/lifecycle warnings are available, that shutdown error includes warning summaries to help tracing-intake setup triage. In-process library snapshots may still be zero-request for inspection.
For TracingTokioSession, runtime snapshot retention also uses the same core capture-limit model. Run metadata time bounds cover merged retained tracing evidence plus retained runtime snapshots, which supports triage interpretation but is not root-cause proof:
- configure retention with
mode(...),capture_limits(...), orcapture_limits_override(...) - there is no tracing-specific
.max_runtime_snapshots(...)session builder method - tracing-only runs still do not fabricate runtime snapshots
§Examples
tailtriage-tracing/examples/live_session_to_run.rstailtriage-tracing/examples/completed_span_jsonl_export.rs
TracingTokioSession::builder(...).run_json_path(...) persists merged Run JSON on shutdown(). Analysis remains a separate tailtriage analyze <run.json> step, and runtime-pressure evidence is triage input rather than root-cause proof.
Tracing intake bridge types for tailtriage triage workflows.
This crate provides semantic tt.* keys, typed SpanRecord intake,
and conversion to tailtriage_core::Run via run_from_span_records.
The jsonl feature adds JSONL import APIs.
The live feature adds live in-memory recording APIs.
It does not implement OpenTelemetry/OTLP. It converts tracing evidence into
standard tailtriage_core::Run artifacts and does not introduce a
tracing-specific analyzer path; Run JSON and the existing analyzer remain the
center of the workflow.
§Example
use tailtriage_tracing::{
ImportOptions, SpanRecord, TT_KIND, TT_OUTCOME, TT_REQUEST_ID, TT_ROUTE,
};
let record = SpanRecord::new("http.request", 1_700_000_000_000, 1_700_000_000_120)
.field(TT_KIND, "request")
.field(TT_REQUEST_ID, "req-42")
.field(TT_ROUTE, "/checkout")
.field(TT_OUTCOME, "ok");
let options = ImportOptions::new("checkout-service").strict(false);
assert_eq!(record.name(), "http.request");
assert_eq!(options.service_name(), "checkout-service");Modules§
- tokio
- Optional Tokio runtime sampler coupling for tracing sessions.
Structs§
- Import
Options - Import options for converting tracing-shaped spans into a run.
- Import
Warning - Non-fatal warning produced while importing tracing-shaped spans.
- Imported
Run - Result of a completed import operation.
- Recorder
Limits - Configurable in-memory limits for live tracing recorder retention.
- Span
Record - A tracing-shaped finished span record ready for intake conversion.
- Tailtriage
Layer tracing_subscriberlayer that feeds completed spans into aTracingRecorder.- Tracing
Intake Session - High-level tracing intake bridge for completed
tt.*spans. - Tracing
Intake Session Builder - Builder for
TracingIntakeSession. - Tracing
Recorder - In-memory recorder for completed tracing spans with
tt.*fields. - Tracing
Recorder Builder - Builder for
TracingRecorder.
Enums§
- Field
Value - Supported scalar field values on imported spans.
- Import
Error - Import failures for tracing-shaped span ingestion.
- Jsonl
Parse Mode - Parse mode for completed tailtriage tracing span JSONL import.
- Span
Kind - Semantic span kind used by tailtriage tracing intake.
Constants§
- DEFAULT_
MAX_ COMPLETED_ CANDIDATE_ SPANS - Default maximum number of closed raw completed candidate spans retained before conversion.
- DEFAULT_
MAX_ OPEN_ SPANS - Default maximum number of concurrently tracked open candidate spans.
- TT_
DEPTH_ AT_ START - Field key for queue depth observed at queue start.
- TT_KIND
- Field key for the span kind (
request,stage, orqueue). - TT_
OUTCOME - Field key for request outcome label.
- TT_
QUEUE - Field key for a queue name.
- TT_
REQUEST_ ID - Field key for a request identifier.
- TT_
ROUTE - Field key for the normalized route name.
- TT_
STAGE - Field key for a stage name.
- TT_
SUCCESS - Field key for boolean stage success.
Functions§
- ensure_
persistable_ run_ has_ requests - Ensures a run is suitable for persisted Run JSON artifacts intended for CLI analysis.
- import_
jsonl_ path - Imports newline-delimited JSON records from a filesystem path.
- import_
jsonl_ path_ with_ mode - Imports newline-delimited JSON records from a filesystem path with an explicit parse mode.
- import_
jsonl_ reader - Imports newline-delimited JSON records from a reader into a converted run.
- import_
jsonl_ reader_ with_ mode - Imports newline-delimited JSON records from a reader with an explicit parse mode.
- run_
from_ span_ records - Converts in-memory tracing span records into a
tailtriage_core::Run.