Skip to main content

Crate tailtriage_tracing

Crate tailtriage_tracing 

Source
Expand description

§tailtriage-tracing

tailtriage-tracing is a narrow tracing intake bridge for completed tt.* spans.

It helps existing tracing users produce standard tailtriage_core::Run artifacts by:

  • writing Run JSON on shutdown, and/or
  • writing retained, semantically valid completed-span JSONL on shutdown.

It is not:

  • an observability backend,
  • generic tracing log scraping,
  • an OTel/OTLP pipeline,
  • proof of root cause (output remains triage leads).

§When to use this crate

Use this path when your service already uses Rust tracing and already has stable per-work-item IDs that can be converted into unique tailtriage request IDs. New integrations without existing tracing/correlation should start with native tailtriage capture first.

This crate converts tracing-shaped request, stage, and queue evidence into standard tailtriage_core::Run artifacts for the normal tailtriage analyze workflow. It is not a tracing backend.

For one completed logical request/work item, every request, stage, and queue span must carry the same tt.request_id. That tt.request_id is the unique tailtriage request ID for the completed request within one Run, not necessarily a raw distributed trace ID. Child stage/queue evidence is correlated to retained request evidence by tt.request_id; missing, inconsistent, or duplicate IDs cause child evidence to be skipped, weakened, or reported as ambiguous.

External trace/correlation IDs may repeat across retries, fanout branches, batch items, or attempts. When they can repeat, derive a unique tailtriage tt.request_id first, for example by adding attempt, span, branch, or item information. Users remain responsible for meaningful instrumentation and request-boundary semantics.

§Feature flags

  • Base crate: typed SpanRecord, ImportOptions, ImportedRun, semantic constants, and run_from_span_records(...).
  • Default (jsonl): JSONL import APIs and stable wrapper parsing.
  • live: enables TracingRecorder, TailtriageLayer, and TracingIntakeSession.
  • tokio: enables TracingTokioSession runtime-sampler coupling and includes live (background sampler on by default; deterministic runs can call disable_background_sampler() and inject snapshots manually).

CLI offline import workflows only need JSONL import support and do not require the live tracing_subscriber layer dependency.

The same APIs are also available through the default tailtriage crate when enabling its tracing, tracing-live, or tracing-tokio façade features.

For live tracing intake sessions, tailtriage-tracing enables optional dependencies behind feature flags, but applications that call tracing::... or tracing_subscriber::... directly still need explicit app dependencies:

cargo add tailtriage-tracing --features live
cargo add tracing tracing-subscriber
cargo add tokio --features macros,rt-multi-thread

If you use Tokio runtime sampler coupling via TracingTokioSession, use:

cargo add tailtriage-tracing --features tokio
cargo add tracing tracing-subscriber
cargo add tokio --features macros,rt-multi-thread
use tailtriage_tracing::TracingIntakeSession;
use tracing::Instrument as _;
use tracing_subscriber::prelude::*;

async fn work() {
    // Your request work goes here.
}

#[tokio::main(flavor = "current_thread")]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let session = TracingIntakeSession::builder("checkout-service")
        .run_json_path("target/tailtriage-examples/checkout.run.json")
        .completed_span_jsonl_path("target/tailtriage-examples/checkout.spans.jsonl")
        .build()?;

    tracing_subscriber::registry()
        .with(session.layer())
        .init(); // startup-only: global subscriber installation for this process

    {
        let request = tracing::info_span!(
            "request",
            tt.kind = "request",
            tt.request_id = "req-1",
            tt.route = "/checkout",
            tt.outcome = "ok"
        );
        work().instrument(request).await;
    } // the request span is closed before shutdown

    let imported = session.shutdown()?;
    let _ = imported;
    Ok(())
}

Install the tailtriage layer beside your existing tracing layers in the application’s normal process-wide subscriber setup; tailtriage augments your tracing pipeline rather than replacing it.

If your service already builds a subscriber in startup code, compose session.layer() onto that subscriber and install it once with your normal global install path (for example .init() during binary startup, or tracing::subscriber::set_global_default(...) with explicit error handling when needed).

set_default is scoped to the current thread and guard lifetime; service startup should install the tailtriage layer in the process-wide subscriber setup.

§Timing model

  • Imported or live tracing evidence is converted to normal tailtriage Run timing fields.
  • Live tracing samples finish wall time when the span closes.
  • Newer live tracing output includes run-relative monotonic offsets for request, stage, and queue spans when those offsets are available; these offsets improve temporal grouping inside a captured run.
  • Imported JSONL may omit run-relative offsets. When they are absent, the analyzer falls back to Unix-ms wall-clock timestamps for temporal grouping.
  • duration_us remains the authoritative elapsed-time evidence when supplied or recorded by live tracing.
  • Completed-span JSONL export preserves duration_us when it is replay-safe under import precedence: complete run-relative monotonic offsets are used when present; otherwise exported Unix-ms bounds are used. When the selected timing source does not match the retained duration, export omits duration_us; replay then derives elapsed time from the best available timing evidence.
  • Unix-ms timestamps are wall-clock anchors and may be coarser than durations.
  • Ordinary tracing log timestamps are not enough for completed-span import; completed spans need explicit start/end timing and semantic tt.* fields.

§Direct Run JSON path

Use run_json_path(...) when you want to skip a separate import step and write Run JSON through the same robust writer path used by native capture sinks:

tailtriage analyze target/tailtriage-examples/checkout.run.json

For both TracingIntakeSession and TracingTokioSession, persisted-output shutdown returns a zero-request error when no request is retained. When tracing intake warnings exist (for example malformed tt.* fields), shutdown includes those warning messages in the same error to guide setup corrections before rerunning capture.

§Completed-span JSONL path

Use completed_span_jsonl_path(...) when you want an offline import workflow:

tailtriage import tracing-spans-jsonl target/tailtriage-examples/checkout.spans.jsonl \
  --service checkout-service \
  --output target/tailtriage-examples/checkout.run.json

tailtriage analyze target/tailtriage-examples/checkout.run.json

If you configure both completed_span_jsonl_path(...) and run_json_path(...), each configured file is written independently through its own temp/rename path. The two outputs are not committed as one atomic transaction: if the second write fails, the first output may already exist as a finalized artifact. For production workflows that need one canonical shutdown artifact, prefer run_json_path(...). Completed-span JSONL remains a replay/debug export, not trace archival.

§Stable JSONL wrapper format

Stable completed-span JSONL records use this wrapper:

{"format":"tailtriage.tracing-span.v1","span":{...}}

format is a wrapper-level field (not a SpanRecord field). The simple library import APIs (import_jsonl_reader / import_jsonl_path) default to this wrapper-only mode and return an error for any non-empty non-wrapper JSON record. Use *_with_mode(..., JsonlParseMode::Compatible) only for pre-stable/internal normalized completed-span records with explicit unix-ms start/end timestamps.

Close-event/fmt-like tracing log envelopes are not supported import input. Ordinary tracing log JSON (for example tracing_subscriber::fmt().json() output) is unsupported and rejected by import. Import does not guess span timing from line receive time: provide explicit unix-ms start/end timestamps on completed spans.

§tt.* field convention

Span kindRequired fieldsOptional fields
requesttt.kind="request", tt.request_id, tt.routett.outcome (optional non-empty string; recommended common labels: ok, error, timeout, cancelled, rejected)
stagett.kind="stage", tt.request_id, tt.stagett.success
queuett.kind="queue", tt.request_id, tt.queuett.depth_at_start

Accepted field types:

  • tt.success: optional bool; strings "true" and "false" are also accepted case-insensitively.
  • tt.depth_at_start: optional non-negative integer. Do not record it with debug formatting.
  • tt.outcome: optional non-empty string.
  • tt.kind, tt.request_id, tt.route, tt.stage, and tt.queue: scalar strings.

Use a plain integer value for queue depth:

tt.depth_at_start = depth_u64

tt.depth_at_start = ?depth may produce a debug-formatted value and be rejected.

For semantic string fields, tt.kind = "request" works, and tt.kind = %kind can work when kind displays as request. tt.kind = ?kind may record "request" with debug quoting and be rejected as unknown.

Missing request tt.outcome defaults to ok with a warning. Missing stage tt.success defaults to true with a warning.

Live tracing intake only tracks spans that are tailtriage candidates at span creation time. Declare tt.* fields when the span is created. If a value is filled later, declare the field with tracing::field::Empty and then call span.record(...); adding brand-new tt.* fields later with span.record(...) is not supported.

let span = tracing::info_span!(
    "request",
    tt.kind = "request",
    tt.request_id = "req-1",
    tt.route = "/checkout",
    tt.outcome = tracing::field::Empty,
);
span.record("tt.outcome", "timeout");

§Strict vs non-strict

  • Strict mode: malformed/incomplete tt.* span records fail import/session conversion.
  • Non-strict mode: malformed/incomplete records are warned and skipped where implemented.
  • Duration consistency rule: complete run-relative offsets are preferred when present; Unix-ms bounds are the fallback when complete run-relative offsets are absent. duration_us remains authoritative elapsed-time evidence when supplied and consistent with the selected timing source. If complete run-relative monotonic offsets are present, strict conversion requires duration_us to match finished_at_run_us - started_at_run_us within 2_000 microseconds. If complete run-relative offsets are absent, strict conversion checks duration_us against (finished_at_unix_ms - started_at_unix_ms) * 1000. Non-strict conversion warns on mismatches but keeps the supplied duration_us. When duration_us is absent, conversion derives elapsed time from complete run-relative offsets first, then falls back to Unix-ms wall-clock bounds.
  • Child stage/queue containment uses a fixed 2 ms tolerance when checking whether child intervals fall inside retained request intervals.
  • That 2 ms containment tolerance is not configurable in this release (no CLI/API knob).

§Retention and drop behavior

  • DEFAULT_MAX_OPEN_SPANS and DEFAULT_MAX_COMPLETED_CANDIDATE_SPANS are live-recorder memory caps for in-memory span tracking.
  • max_open_spans bounds in-flight span tracking.
  • max_completed_candidate_spans is a raw live-recorder memory cap for closed candidates before semantic conversion.
  • Under raw-cap pressure, request roots are preserved preferentially when possible.
  • Child stage/queue evidence may be dropped or evicted under that pressure and is surfaced through warnings plus truncation.limits_hit.
  • Request/stage/queue semantic retention uses CaptureMode, CaptureLimits, and CaptureLimitsOverride.
  • completed_span_jsonl_path(...) writes retained tailtriage semantic evidence as stable span-shaped JSONL on shutdown only when at least one completed request is retained.
  • Completed-span JSONL is retained-evidence replay/debug export for the same request/stage/queue evidence path through tailtriage import; it is not a production trace archive.
  • It does not preserve lifecycle warnings, truncation counters, original span IDs, parent IDs, original span names, or non-tt.* fields.
  • For production workflows that need the complete persisted triage artifact including warnings/truncation metadata, prefer run_json_path(...).
  • Callers using JSONL-only export should inspect session.shutdown()?.warnings() in the same process.
  • This completed-span JSONL is a narrow retained-evidence export, not a generic tracing log stream and not OTel/OTLP.

§Runtime-pressure limitation

Tracing intake import and native capture share the same CaptureMode/CaptureLimits semantics for request/stage/queue evidence retention. Offline completed tailtriage tracing span JSONL import does not fabricate runtime snapshots. Runtime-pressure evidence still requires runtime snapshots/Tokio sampler coupling. Runtime-sensitive tracing contract parity uses deterministic/manual runtime snapshots and requires non-empty runtime snapshots, scenario-specific runtime field evidence, and the explicit disabled-background-sampler lifecycle warning (via disable_background_sampler() + record_runtime_snapshot(...)). It does not rely on ambient sampler metadata/noise. Persisted Run JSON intended for tailtriage analyze must include at least one completed request event; shutdown fails for persisted-output sessions when zero completed requests are retained. When intake/lifecycle warnings are available, that shutdown error includes warning summaries to help tracing-intake setup triage. In-process library snapshots may still be zero-request for inspection.

For TracingTokioSession, runtime snapshot retention also uses the same core capture-limit model. Run metadata time bounds cover merged retained tracing evidence plus retained runtime snapshots, which supports triage interpretation but is not root-cause proof:

  • configure retention with mode(...), capture_limits(...), or capture_limits_override(...)
  • there is no tracing-specific .max_runtime_snapshots(...) session builder method
  • tracing-only runs still do not fabricate runtime snapshots

§Examples

  • tailtriage-tracing/examples/live_session_to_run.rs
  • tailtriage-tracing/examples/completed_span_jsonl_export.rs

TracingTokioSession::builder(...).run_json_path(...) persists merged Run JSON on shutdown(). Analysis remains a separate tailtriage analyze <run.json> step, and runtime-pressure evidence is triage input rather than root-cause proof. Tracing intake bridge types for tailtriage triage workflows.

This crate provides semantic tt.* keys, typed SpanRecord intake, and conversion to tailtriage_core::Run via run_from_span_records. The jsonl feature adds JSONL import APIs. The live feature adds live in-memory recording APIs. It does not implement OpenTelemetry/OTLP. It converts tracing evidence into standard tailtriage_core::Run artifacts and does not introduce a tracing-specific analyzer path; Run JSON and the existing analyzer remain the center of the workflow.

§Example

use tailtriage_tracing::{
    ImportOptions, SpanRecord, TT_KIND, TT_OUTCOME, TT_REQUEST_ID, TT_ROUTE,
};

let record = SpanRecord::new("http.request", 1_700_000_000_000, 1_700_000_000_120)
    .field(TT_KIND, "request")
    .field(TT_REQUEST_ID, "req-42")
    .field(TT_ROUTE, "/checkout")
    .field(TT_OUTCOME, "ok");

let options = ImportOptions::new("checkout-service").strict(false);
assert_eq!(record.name(), "http.request");
assert_eq!(options.service_name(), "checkout-service");

Modules§

tokio
Optional Tokio runtime sampler coupling for tracing sessions.

Structs§

ImportOptions
Import options for converting tracing-shaped spans into a run.
ImportWarning
Non-fatal warning produced while importing tracing-shaped spans.
ImportedRun
Result of a completed import operation.
RecorderLimits
Configurable in-memory limits for live tracing recorder retention.
SpanRecord
A tracing-shaped finished span record ready for intake conversion.
TailtriageLayer
tracing_subscriber layer that feeds completed spans into a TracingRecorder.
TracingIntakeSession
High-level tracing intake bridge for completed tt.* spans.
TracingIntakeSessionBuilder
Builder for TracingIntakeSession.
TracingRecorder
In-memory recorder for completed tracing spans with tt.* fields.
TracingRecorderBuilder
Builder for TracingRecorder.

Enums§

FieldValue
Supported scalar field values on imported spans.
ImportError
Import failures for tracing-shaped span ingestion.
JsonlParseMode
Parse mode for completed tailtriage tracing span JSONL import.
SpanKind
Semantic span kind used by tailtriage tracing intake.

Constants§

DEFAULT_MAX_COMPLETED_CANDIDATE_SPANS
Default maximum number of closed raw completed candidate spans retained before conversion.
DEFAULT_MAX_OPEN_SPANS
Default maximum number of concurrently tracked open candidate spans.
TT_DEPTH_AT_START
Field key for queue depth observed at queue start.
TT_KIND
Field key for the span kind (request, stage, or queue).
TT_OUTCOME
Field key for request outcome label.
TT_QUEUE
Field key for a queue name.
TT_REQUEST_ID
Field key for a request identifier.
TT_ROUTE
Field key for the normalized route name.
TT_STAGE
Field key for a stage name.
TT_SUCCESS
Field key for boolean stage success.

Functions§

ensure_persistable_run_has_requests
Ensures a run is suitable for persisted Run JSON artifacts intended for CLI analysis.
import_jsonl_path
Imports newline-delimited JSON records from a filesystem path.
import_jsonl_path_with_mode
Imports newline-delimited JSON records from a filesystem path with an explicit parse mode.
import_jsonl_reader
Imports newline-delimited JSON records from a reader into a converted run.
import_jsonl_reader_with_mode
Imports newline-delimited JSON records from a reader with an explicit parse mode.
run_from_span_records
Converts in-memory tracing span records into a tailtriage_core::Run.