Skip to main content

Crate init_tracing_opentelemetry

Crate init_tracing_opentelemetry 

Source
Expand description

§init-tracing-opentelemetry

crates license crate version

A set of helpers to initialize (and more) tracing + opentelemetry (compose your own or use opinionated preset)

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    // Simple preset
    let _guard = init_tracing_opentelemetry::TracingConfig::production().init_subscriber()?;

    //...

    Ok(())
}
#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    // custom configuration
    let _guard = init_tracing_opentelemetry::TracingConfig::default()
        .with_json_format()
        .with_stderr()
        .with_log_directives("debug")
        .init_subscriber()?;

    //...

    Ok(())
}

The init_subscriber() function returns an OtelGuard instance. Following the guard pattern, this struct provides no functions but, when dropped, ensures that any pending traces/metrics are sent before it exits. The syntax let _guard is suggested to ensure that Rust does not drop the struct until the application exits.

§Configuration Options

§Presets

  • TracingConfig::development() - Pretty format, stderr, with debug info
  • TracingConfig::production() - JSON format, stdout, minimal metadata
  • TracingConfig::debug() - Full verbosity with all span events
  • TracingConfig::minimal() - Compact format, no OpenTelemetry
  • TracingConfig::testing() - Minimal output for tests

§Custom Configuration

use init_tracing_opentelemetry::TracingConfig;

TracingConfig::default()
    .with_pretty_format()           // or .with_json_format(), .with_compact_format()
    .with_stderr()                  // or .with_stdout(), .with_file(path)
    .with_log_directives("debug")   // Custom log levels
    .with_line_numbers(true)        // Include line numbers
    .with_thread_names(true)        // Include thread names
    .with_otel(true)                // Enable OpenTelemetry
    .init_subscriber()
    .expect("valid tracing configuration");

§Add custom layer, modify subscriber

Use init_subscriber_ext(|subscriber| {...} ) to transform the subscriber (registry), before application of the configuration.

use init_tracing_opentelemetry::TracingConfig;
use tokio_blocked::TokioBlockedLayer;
use tracing::info;
use tracing_subscriber::layer::SubscriberExt;

#[tokio::main]
async fn main() {
    let blocked = TokioBlockedLayer::new()
        .with_warn_busy_single_poll(Some(std::time::Duration::from_micros(150)));

    let _guard = TracingConfig::default()
        .with_log_directives("info,tokio::task=trace,tokio::task::waker=warn")
        .with_span_events(tracing_subscriber::fmt::format::FmtSpan::NONE)
        .init_subscriber_ext(|subscriber| subscriber.with(blocked))
        .unwrap();

    tokio::task::spawn(async {
        // BAD!
        // This produces a warning log message.
        info!("blocking!");
        std::thread::sleep(std::time::Duration::from_secs(1));
    })
    .await
    .unwrap();

    tokio::time::sleep(tokio::time::Duration::from_secs(5)).await;
}

§Legacy API (deprecated)

For backward compatibility, the old API is still available:

pub fn build_loglevel_filter_layer() -> tracing_subscriber::filter::EnvFilter {
    // filter what is output on log (fmt)
    // std::env::set_var("RUST_LOG", "warn,axum_tracing_opentelemetry=info,otel=debug");
    std::env::set_var(
        "RUST_LOG",
        format!(
            // `otel::tracing` should be a level trace to emit opentelemetry trace & span
            // `otel::setup` set to debug to log detected resources, configuration read and infered
            "{},otel::tracing=trace,otel=debug",
            std::env::var("RUST_LOG")
                .or_else(|_| std::env::var("OTEL_LOG_LEVEL"))
                .unwrap_or_else(|_| "info".to_string())
        ),
    );
    EnvFilter::from_default_env()
}

pub fn build_otel_layer<S>() -> Result<OpenTelemetryLayer<S, Tracer>, BoxError>
where
    S: Subscriber + for<'a> LookupSpan<'a>,
{
    use crate::{
        init_propagator, //stdio,
        otlp,
        resource::DetectResource,
    };
    let otel_rsrc = DetectResource::default()
        //.with_fallback_service_name(env!("CARGO_PKG_NAME"))
        //.with_fallback_service_version(env!("CARGO_PKG_VERSION"))
        .build();
    let otel_tracer = otlp::init_tracer(otel_rsrc, otlp::identity)?;
    // to not send trace somewhere, but continue to create and propagate,...
    // then send them to `axum_tracing_opentelemetry::stdio::WriteNoWhere::default()`
    // or to `std::io::stdout()` to print
    //
    // let otel_tracer =
    //     stdio::init_tracer(otel_rsrc, stdio::identity, stdio::WriteNoWhere::default())?;
    init_propagator()?;
    Ok(tracing_opentelemetry::layer().with_tracer(otel_tracer))
}

To retrieve the current trace_id (eg to add it into error message (as header or attributes))


  let trace_id = tracing_opentelemetry_instrumentation_sdk::find_current_trace_id();
  //json!({ "error" :  "xxxxxx", "trace_id": trace_id})

§Configuration based on the environment variables

To ease setup and compliance with OpenTelemetry SDK configuration, the configuration can be done with the following environment variables (see sample init_tracing() above):

  • OTEL_EXPORTER_OTLP_TRACES_ENDPOINT fallback to OTEL_EXPORTER_OTLP_ENDPOINT for the url of the exporter / collector
  • OTEL_EXPORTER_OTLP_TRACES_PROTOCOL fallback to OTEL_EXPORTER_OTLP_PROTOCOL, fallback to auto-detection based on ENDPOINT port
  • OTEL_SERVICE_NAME for the name of the service
  • OTEL_PROPAGATORS for the configuration of the propagators
  • OTEL_TRACES_SAMPLER & OTEL_TRACES_SAMPLER_ARG for configuration of the sampler

Few other environment variables can also be used to configure OTLP exporter (eg to configure headers, authentication,, etc…):

# For GRPC:
export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT="http://localhost:4317"
export OTEL_EXPORTER_OTLP_TRACES_PROTOCOL="grpc"
export OTEL_TRACES_SAMPLER="always_on"

# For HTTP:
export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT="http://127.0.0.1:4318/v1/traces"
export OTEL_EXPORTER_OTLP_TRACES_PROTOCOL="http/protobuf"
export OTEL_TRACES_SAMPLER="always_on"

In the context of kubernetes, some of the above environment variables can be injected by the Opentelemetry operator (via inject-sdk):

apiVersion: apps/v1
kind: Deployment
spec:
  template:
    metadata:
      annotations:
        # to inject environment variables only by opentelemetry-operator
        instrumentation.opentelemetry.io/inject-sdk: "opentelemetry-operator/instrumentation"
        instrumentation.opentelemetry.io/container-names: "app"
      containers:
        - name: app

Or if you don’t setup inject-sdk, you can manually set the environment variable eg

apiVersion: apps/v1
kind: Deployment
spec:
  template:
    metadata:
      containers:
        - name: app
          env:
            - name: OTEL_SERVICE_NAME
              value: "app"
            - name: OTEL_EXPORTER_OTLP_PROTOCOL
              value: "grpc"
            # for otel collector in `deployment` mode, use the name of the service
            # - name: OTEL_EXPORTER_OTLP_ENDPOINT
            #   value: "http://opentelemetry-collector.opentelemetry-collector:4317"
            # for otel collector in sidecar mode (imply to deploy a sidecar CR per namespace)
            - name: OTEL_EXPORTER_OTLP_ENDPOINT
              value: "http://localhost:4317"
            # for `daemonset` mode: need to use the local daemonset (value interpolated by k8s: `$(...)`)
            # - name: OTEL_EXPORTER_OTLP_ENDPOINT
            #   value: "http://$(HOST_IP):4317"
            # - name: HOST_IP
            #   valueFrom:
            #     fieldRef:
            #       fieldPath: status.hostIP

§Troubleshot why no trace?

  • check you only have a single version of opentelemtry (could be part of your CI/build), use cargo-deny or cargo tree

    # Check only one version of opentelemetry should be used
# else issue with setup of global (static variable)
# check_single_version_opentelemtry:
cargo tree -i opentelemetry

  • check the code of your exporter and the integration with tracing (as subscriber’s layer)

  • check the environment variables of opentelemetry OTEL_EXPORTER... and OTEL_TRACES_SAMPLER (values are logged on target otel::setup )

  • check that log target otel::tracing enable log level trace (or info if you use tracing_level_info feature) to generate span to send to opentelemetry collector.

§Metrics

To configure opentelemetry metrics, enable the metrics feature, this will initialize a SdkMeterProvider, set it globally and add a a MetricsLayer to allow using tracing events to produce metrics.

The opentelemetry_sdk can still be used to produce metrics as well, since we configured the SdkMeterProvider globally, so any Axum/Tonic middleware that does not use tracing but directly opentelemetry::metrics will work.

Configure the following set of environment variables to configure the metrics exporter (on top of those configured above):

  • OTEL_EXPORTER_OTLP_METRICS_ENDPOINT override to OTEL_EXPORTER_OTLP_ENDPOINT for the url of the exporter / collector
  • OTEL_EXPORTER_OTLP_METRICS_PROTOCOL override to OTEL_EXPORTER_OTLP_PROTOCOL, fallback to auto-detection based on ENDPOINT port
  • OTEL_EXPORTER_OTLP_METRICS_TIMEOUT to set the timeout for the connection to the exporter
  • OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE to set the temporality preference for the exporter
  • OTEL_METRIC_EXPORT_INTERVAL to set frequence of metrics export in milliseconds, defaults to 60s

§Logs

To configure OpenTelemetry log export, enable the logs feature. This initializes a SdkLoggerProvider and adds a log bridge layer so that tracing events are forwarded to the OpenTelemetry log pipeline and exported via OTLP.

[dependencies]
init-tracing-opentelemetry = { version = "*", features = ["otlp", "logs"] }

Standard tracing macros emit logs that are automatically bridged:

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let _guard = init_tracing_opentelemetry::TracingConfig::production().init_subscriber()?;

    tracing::error!("This is ground control to Major Tom");
    tracing::warn!("Houston, we have a problem");
    tracing::info!("We have contact");
    tracing::debug!("Roger, copy that");

    Ok(())
}

Log export can be toggled at runtime via .with_logs(bool):

use init_tracing_opentelemetry::TracingConfig;
//...
TracingConfig::default()
    .with_logs(false)   // disable log export (default: enabled when feature is active)
    .init_subscriber()
    .expect("valid tracing configuration");

Traces are automatically attached to logs as well, so if the logs are queried in Grafana (for example), the trace automatically links to the log line.

screenshot of grafana logs

Configure the following environment variables to control the logs exporter (in addition to the shared variables above):

  • OTEL_EXPORTER_OTLP_LOGS_ENDPOINT overrides OTEL_EXPORTER_OTLP_ENDPOINT for the log pipeline; for HTTP the path /v1/logs is appended automatically
  • OTEL_EXPORTER_OTLP_LOGS_PROTOCOL overrides OTEL_EXPORTER_OTLP_PROTOCOL, falls back to port-based auto-detection
# For GRPC:
export OTEL_EXPORTER_OTLP_LOGS_ENDPOINT="http://localhost:4317"
export OTEL_EXPORTER_OTLP_LOGS_PROTOCOL="grpc"

# For HTTP:
export OTEL_EXPORTER_OTLP_LOGS_ENDPOINT="http://127.0.0.1:4318/v1/logs"
export OTEL_EXPORTER_OTLP_LOGS_PROTOCOL="http/protobuf"

Note: A protocol must be set (via env var or inferable from the endpoint port). If neither is found, no log exporter is created and a warning is emitted on target otel::setup.

§Changelog - History

CHANGELOG.md

Re-exports§

pub use config::FeatureSet;tracing_subscriber_ext
pub use config::Guard;tracing_subscriber_ext
pub use config::LevelConfig;tracing_subscriber_ext
pub use config::LogFormat;tracing_subscriber_ext
pub use config::LogTimer;tracing_subscriber_ext
pub use config::OtelConfig;tracing_subscriber_ext
pub use config::TracingConfig;tracing_subscriber_ext
pub use config::WriterConfig;tracing_subscriber_ext
pub use formats::CompactLayerBuilder;tracing_subscriber_ext
pub use formats::FullLayerBuilder;tracing_subscriber_ext
pub use formats::JsonLayerBuilder;tracing_subscriber_ext
pub use formats::LayerBuilder;tracing_subscriber_ext
pub use formats::PrettyLayerBuilder;tracing_subscriber_ext
pub use formats::LogfmtLayerBuilder;tracing_subscriber_ext and logfmt
pub use opentelemetry_sdk;
pub use tracing_opentelemetry;

Modules§

configtracing_subscriber_ext
Flexible tracing configuration with builder pattern.
formatstracing_subscriber_ext
Format-specific layer builders for tracing output.
otlpotlp
resourcetracer
stdiostdout
tracing_subscriber_exttracing_subscriber_ext

Enums§

Error

Functions§

init_propagator
Configure the global propagator based on content of the env variable OTEL_PROPAGATORS Specifies Propagators to be used in a comma-separated list. Default value: "tracecontext,baggage" Example: export OTEL_PROPAGATORS="b3" Accepted values for OTEL_PROPAGATORS are: