Expand description
Everything needed to set up telemetry (logs, traces, metrics) for both clients and servers.
Despite the name re_perf_telemetry, this actually handles all forms of telemetry,
including all log output.
This sort of telemetry is always disabled on our OSS binaries, and is only used for
- The Rerun Hub infrastructure
- Profiling by Rerun developer
§Logging strategy
-
All our logs go through the structured
tracingmacros. -
We always log from
tracingdirectly into stdio: we never involve theOpenTelemetrylogging API. Production is expected to read the logs from the pod’s output. There is never any internal buffering going on, besides the buffering of stdio itself. -
All logs that happen as part of the larger trace/span will automatically be uploaded with that trace/span. This makes our traces a very powerful debugging tool, in addition to a profiler.
§Tracing strategy
-
All our traces go through the structured
tracingmacros. We never use theOpenTelemetrymacros. -
The traces go through a first layer of filtering based on the value of
RUST_TRACE, which functions similarly to aRUST_LOGfilter. -
The traces are then sent to the
OpenTelemetrySDK, where they will go through a pass of sampling before being sent to the OTLP endpoint. The sampling mechanism is controlled by the official OTEL environment variables. -
Spans that contains error logs will properly be marked as failed, and easily findable.
§Metric strategy
-
Our metric strategy is basically the opposite of our logging strategy: everything goes through
OpenTelemetrydirectly,tracingis never involved. -
Metrics are uploaded (as opposed to scrapped!) using the OTLP protocol, on a fixed interval defined by the
OTEL_METRIC_EXPORT_INTERVALenvironment variable.
Modules§
Structs§
- Client
OnResponse - Implements a
tower_http::trace::OnResponsemiddleware for the gRPC client. - Grpc
Make Span - Implements
tower_http::trace::MakeSpanwhere the trace name is the gRPC method name. - Grpc
OnEos - Implements a
tower_http::trace::OnEosmiddleware. - Grpc
OnFirst Body Chunk - Implements a
tower_http::trace::OnBodyChunkmiddleware, but only accounts for the first one. - Grpc
OnRequest - Implements a
tower_http::trace::OnRequestmiddleware. - Grpc
OnResponse - Implements a
tower_http::trace::OnResponsemiddleware. - Grpc
OnResponse Options - Rerun
Tracing Session Id - A validated rerun session id.
- Span
Metadata Cleanup Layer - A
tracing_subscriber::Layerthat cleans upSpanMetadataentries when spans close. - Telemetry
- The Redap telemetry pipeline.
- Telemetry
Args - Complete configuration for all things telemetry.
- Telemetry
Layer Options - Trace
Headers - Tracing
Injector Interceptor - This implements a
tonic::service::Interceptorthat injects trace/span metadata into the request headers, according to W3C standards.
Enums§
Constants§
- RERUN_
SESSION_ TRACESTATE_ KEY - The W3C
tracestatekey under which the rerun session id propagates.
Traits§
Functions§
- current_
rerun_ session_ id - Returns the active rerun session id, if any.
- current_
trace_ headers - Export the active trace in the current context as the W3C trace headers, if any.
- current_
trace_ id - Returns the active
TraceIdin the current context, if any. - dec_
active_ tracing_ session_ count - Decrement the active-session counter. Called by Python
tracing_session().__exit__; the Rustwith_tracing_sessionpath goes throughActiveSessionGuard’s drop instead. - inc_
active_ tracing_ session_ count - Increment the active-session counter. Called by Python
tracing_session().__enter__; the Rustwith_tracing_sessionpath goes throughActiveSessionGuardinstead. - is_
telemetry_ active - Returns
trueonceTelemetry::inithas run with telemetry enabled (i.e. thetracingsubscriber, OTLP exporters, and global propagator are installed). - new_
client_ telemetry_ layer - Creates a new
tower::Layermiddleware that automatically: - new_
server_ telemetry_ layer - Creates a new
tower::Layermiddleware that automatically: - to_
short_ str - Displays a possibly long list of items to a length limited list of items.
- with_
current_ tracing_ session - Wrap
fso the active rerun session id is resolved once at entry (via the registered [SessionIdReader]) and stays accessible to every outbound gRPC inside it without re-invoking the reader. - with_
tracing_ session - Tag every Rerun Hub request inside
fwith a fresh session id, so the full set of requests can be correlated end-to-end for support.