Crate re_perf_telemetry

Expand description

Everything needed to set up telemetry (logs, traces, metrics) for both clients and servers.

Despite the name re_perf_telemetry, this actually handles all forms of telemetry, including all log output.

This sort of telemetry is always disabled on our OSS binaries, and is only used for

§Logging strategy

All our logs go through the structured tracing macros.
We always log from tracing directly into stdio: we never involve the OpenTelemetry logging API. Production is expected to read the logs from the pod’s output. There is never any internal buffering going on, besides the buffering of stdio itself.
All logs that happen as part of the larger trace/span will automatically be uploaded with that trace/span. This makes our traces a very powerful debugging tool, in addition to a profiler.

All our traces go through the structured tracing macros. We never use the OpenTelemetry macros.
The traces go through a first layer of filtering based on the value of RUST_TRACE, which functions similarly to a RUST_LOG filter.
The traces are then sent to the OpenTelemetry SDK, where they will go through a pass of sampling before being sent to the OTLP endpoint. The sampling mechanism is controlled by the official OTEL environment variables.
Spans that contains error logs will properly be marked as failed, and easily findable.

Our metric strategy is basically the opposite of our logging strategy: everything goes through OpenTelemetry directly, tracing is never involved.
Metrics are uploaded (as opposed to scrapped!) using the OTLP protocol, on a fixed interval defined by the OTEL_METRIC_EXPORT_INTERVAL environment variable.

GrpcMakeSpan: Implements tower_http::trace::MakeSpan where the trace name is the gRPC method name.
GrpcOnEos: Implements a tower_http::trace::OnEos middleware.
GrpcOnFirstBodyChunk: Implements a tower_http::trace::OnBodyChunk middleware, but only accounts for the first one.
GrpcOnRequest: Implements a tower_http::trace::OnRequest middleware.
GrpcOnResponse: Implements a tower_http::trace::OnResponse middleware.
GrpcOnResponseOptions
Telemetry: The Redap telemetry pipeline.
TelemetryArgs: Complete configuration for all things telemetry.
TelemetryLayerOptions
TraceHeaders
TraceIdLayer: A tracing_subscriber::Layer that injects the opentelemetry trace_id as a benchmark_id field top level field on every span.
TracingInjectorInterceptor: This implements a tonic::service::Interceptor that injects trace/span metadata into the request headers, according to W3C standards.

TRACE_CONTEXT_VAR_NAME: The name of the ContextVar used for trace context propagation

current_trace_headers: Export the active trace in the current context as the W3C trace headers, if any.
current_trace_id: Returns the active TraceId in the current context, if any.
new_client_telemetry_layer: Creates a new tower::Layer middleware that automatically:
new_server_telemetry_layer: Creates a new tower::Layer middleware that automatically:
to_short_str: Displays a possibly long list of items to a length limited list of items.