turnframe-telemetry

Metrics, structured tracing and a reliability dashboard for a Turnframe runtime. Implementations of turnframe_core::observe::Observer you can hand to the runtime at start-up, plus the data that turns them into a board.

Part of the Turnframe workspace. See the workspace README and architecture guide.

Scope

LLM observability backends

Provider calls are described with the OpenTelemetry GenAI semantic conventions (gen_ai.system, gen_ai.operation.name, request and response model, temperature, response id, finish reasons, token usage), which is what Langfuse, Datadog LLM Observability, Phoenix and Braintrust already consume. No vendor is hardcoded: a backend that wants its own spelling is a five-line renaming function over TraceGrouping::fields() or ProviderCall::attributes() in the adopter's code.

Token accounting is explicit, because it is the part that is easy to get wrong: gen_ai.usage.input_tokens is recorded net of gen_ai.usage.input_cached_tokens. Read the first alone for the uncached cost, add both for the whole prompt; neither double counts. ProviderCall::with_cached_usage(total, cached, output) does the subtraction for a provider that reports a total.

TraceGrouping carries the session (the conversation), a hashed end-user reference, tags, environment and release. These backends filter at the level of the individual span, not only at the trace root, so the grouping has to reach every span. Without the otel feature, TraceGrouping::stamp fills it in on any span this crate opened. With the otel feature it is automatic: register otel::GroupingSpanProcessor on your tracer provider and take an otel::attach_grouping guard at the top of the turn.

let provider = SdkTracerProvider::builder()
    .with_span_processor(GroupingSpanProcessor::new())
    .with_span_processor(your_exporting_processor)
    .build();

let _guard = attach_grouping(&grouping);
// every span opened from here on carries session.id, user.id, tags,
// deployment.environment.name and service.version, children included.

Ordering matters. on_start reads the context at the moment a span begins, so the grouping must be attached before the spans that should carry it are created; a span already underway cannot be back-filled. Only the documented grouping keys are copied, so an application's own baggage never leaks into telemetry.

Conversation content is off by default

Prompts and completions are user data, and this crate does not record them unless you say so. ContentRecorder::disabled() is the default; switching it on takes ContentRecorder::enabled(redactor) with a ContentRedactor you wrote, and every string passes through that hook. Plainly: turning content recording on sends what your users typed, and what the model answered, to your tracing backend. Treat it as a data-residency and consent decision, not a debugging convenience.

The label rule

No user text and no case text ever becomes a metric label. Three things enforce it:

1. A label value can only come from a typed field of SignalLabels: workflow key, provider key, model key, request purpose, risk class, interaction kind and a stable error code. There is no free-string field to abuse.
2. Every signal declares the dimensions it documents; anything else a caller attached is dropped, so one call site cannot inflate a metric's cardinality.
3. Every surviving value must pass is_safe_label_value: not empty, at most 64 bytes, no whitespace, and not a UUID. A record identifier therefore cannot reach a metric dimension even when an application puts one in an error code by mistake.

Identifiers belong in traces instead, and even there the account id is hashed rather than logged (tracing::account_hash).

Minimal example

use std::sync::Arc;
use std::time::Duration;

use turnframe_core::ids::WorkflowKey;
use turnframe_core::observe::{Observer, Signal, SignalLabels};
use turnframe_telemetry::{CompositeObserver, MetricsObserver, TracingObserver, metrics};

// Once, after the metrics exporter is installed.
metrics::describe_all();

let observer: Arc<dyn Observer> = Arc::new(
    CompositeObserver::new()
        .with(MetricsObserver::new())
        .with(TracingObserver::new()),
);

let labels = SignalLabels::workflow(WorkflowKey::from("trip"));
observer.observe_labeled(&Signal::TurnReceived, &labels);
observer.observe_duration(&Signal::TurnDuration, Duration::from_millis(412), &labels);

Metric catalogue

Generated from the code by metrics::catalogue_markdown(); a test regenerates it and fails when this table drifts.

Reliability dashboard

Dashboard::reliability() describes the seven panels of spec §26.3 (side-effect integrity failures, claim integrity failures, semantic understanding failures, clarification rate, abandonment rate, provider failures and user experience scores) and names the metrics that feed each one. It is plain serializable data, so the same description can drive a Grafana or Datadog board, an alert inventory, or a docs page through Dashboard::to_markdown().

The split is the point: a single "agent accuracy" percentage would average a misworded sentence together with a wrongly rebooked flight, and nobody can act on that number.

License

Licensed under either of Apache License, Version 2.0 or MIT license at your option.