axum-tracing-opentelemetry 0.8.0

Middlewares and tools to integrate axum + tracing + opentelemetry
Documentation

axum-tracing-opentelemetry

crates license crate version

Project Status: Active – The project has reached a stable, usable state and is being actively developed.

Middlewares and tools to integrate axum + tracing + opentelemetry.

  • Read OpenTelemetry header from incoming request
  • Start a new trace if no trace found in the incoming request
  • Trace is attached into tracing'span

For examples, you can look at:

//...
use axum_tracing_opentelemetry::opentelemetry_tracing_layer;

fn init_tracing() -> Result<(), Box<dyn std::error::Error>> {
    use tracing_subscriber::filter::EnvFilter;
    use tracing_subscriber::fmt::format::FmtSpan;
    use tracing_subscriber::layer::SubscriberExt;

    let subscriber = tracing_subscriber::registry();

    // register opentelemetry tracer layer
    let otel_layer = {
        use axum_tracing_opentelemetry::{
            init_propagator, //stdio,
            make_resource,
            otlp,
        };
        let otel_rsrc = make_resource(
            std::env::var("OTEL_SERVICE_NAME")
                .unwrap_or_else(|_| env!("CARGO_PKG_NAME").to_string()),
            env!("CARGO_PKG_VERSION"),
        );
        let otel_tracer = otlp::init_tracer(otel_rsrc, otlp::identity).expect("setup of Tracer");
        // let otel_tracer =
        //     stdio::init_tracer(otel_rsrc, stdio::identity, stdio::WriteNoWhere::default())
        //         .expect("setup of Tracer");

        // init propagator based on OTEL_PROPAGATORS value
        init_propagator()?;
        tracing_opentelemetry::layer().with_tracer(otel_tracer)
    };
    let subscriber = subscriber.with(otel_layer);

    // filter what is output on log (fmt), but not what is send to trace (opentelemetry collector)
    // std::env::set_var("RUST_LOG", "info,kube=trace");
    std::env::set_var(
        "RUST_LOG",
        std::env::var("RUST_LOG")
            .or_else(|_| std::env::var("OTEL_LOG_LEVEL"))
            .unwrap_or_else(|_| "info".to_string()),
    );
    let subscriber = subscriber.with(EnvFilter::from_default_env());

    if cfg!(debug_assertions) {
        let fmt_layer = tracing_subscriber::fmt::layer()
            .pretty()
            .with_line_number(true)
            .with_thread_names(true)
            .with_span_events(FmtSpan::NEW | FmtSpan::CLOSE)
            .with_timer(tracing_subscriber::fmt::time::uptime());
        let subscriber = subscriber.with(fmt_layer);
        tracing::subscriber::set_global_default(subscriber)?;
    } else {
        let fmt_layer = tracing_subscriber::fmt::layer()
            .json()
            .with_span_events(FmtSpan::NEW | FmtSpan::CLOSE)
            .with_timer(tracing_subscriber::fmt::time::uptime());
        let subscriber = subscriber.with(fmt_layer);
        tracing::subscriber::set_global_default(subscriber)?;
    };
    Ok(())
}

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    init_tracing()?;
    let app = app();
    // run it
    let addr = &"0.0.0.0:3000".parse::<SocketAddr>()?;
    tracing::warn!("listening on {}", addr);
    axum::Server::bind(&addr)
        .serve(app.into_make_service())
        .with_graceful_shutdown(shutdown_signal())
        .await?;
    Ok(())
}

fn app() -> Router {
    // build our application with a route
    Router::new()
        .route("/", get(health)) // request processed inside span
        // opentelemetry_tracing_layer setup `TraceLayer`, that is provided by tower-http so you have to add that as a dependency.
        .layer(opentelemetry_tracing_layer())
        .route("/health", get(health)) // request processed without span / trace
}

async fn shutdown_signal() {
    //...
    opentelemetry::global::shutdown_tracer_provider();
}

To retrieve the current trace_id (eg to add it into error message (as header or attributes))

  let trace_id = axum_tracing_opentelemetry::find_current_trace_id();
  json!({ "error" :  "xxxxxx", "trace_id": trace_id})

To also inject the trace id into the response (could be useful for debugging) uses the layer response_with_trace_layer

    // build our application with a route
    Router::new()
        ...
        // include trace context as header into the response
        .layer(response_with_trace_layer())

Configuration based on environment variable

To ease setup and compliancy with Opentelemetry SDK configuration, the configuration can be done with the following environment variables (see sample init_tracing() above):

  • OTEL_EXPORTER_OTLP_TRACES_ENDPOINT fallback to OTEL_EXPORTER_OTLP_ENDPOINT for the url of the exporter / collector
  • OTEL_EXPORTER_OTLP_TRACES_PROTOCOL fallback to OTEL_EXPORTER_OTLP_PROTOCOL, fallback to autodetection based on ENDPOINT port
  • OTEL_SERVICE_NAME for the name of the service
  • OTEL_PROPAGATORS for the configuration of propagator

In the context of kubernetes, the above environment variable can be injected by the Opentelemetry operator (via inject-sdk):

apiVersion: apps/v1
kind: Deployment
spec:
  template:
    metadata:
      annotations:
        # to inject environment variables only by opentelemetry-operator
        instrumentation.opentelemetry.io/inject-sdk: "opentelemetry-operator/instrumentation"
        instrumentation.opentelemetry.io/container-names: "app"
      containers:
        - name: app

Or if you don't setup inject-sdk, you can manually set the environment variable eg

apiVersion: apps/v1
kind: Deployment
spec:
  template:
    metadata:
      containers:
        - name: app
          env:
            - name: OTEL_SERVICE_NAME
              value: "app"
            - name: OTEL_EXPORTER_OTLP_PROTOCOL
              value: "grpc"
            # for otel collector in `deployment` mode, use the name of the service
            # - name: OTEL_EXPORTER_OTLP_ENDPOINT
            #   value: "http://opentelemetry-collector.opentelemetry-collector:4317"
            # for otel collector in sidecar mode (imply to deploy a sidecar CR per namespace)
            - name: OTEL_EXPORTER_OTLP_ENDPOINT
              value: "http://localhost:4317"
            # for `daemonset` mode: need to use the local daemonset (value interpolated by k8s: `$(...)`)
            # - name: OTEL_EXPORTER_OTLP_ENDPOINT
            #   value: "http://$(HOST_IP):4317"
            # - name: HOST_IP
            #   valueFrom:
            #     fieldRef:
            #       fieldPath: status.hostIP

examples/otlp

In a terminal, run

❯ cd examples/otlp
❯ cargo run
   Compiling examples-otlp v0.1.0 (/home/david/src/github.com/davidB/axum-tracing-opentelemetry/examples/otlp)
    Finished dev [unoptimized + debuginfo] target(s) in 2.96s
     Running `target/debug/examples-otlp`
     0.000170750s  WARN examples_otlp: listening on 0.0.0.0:3003
    at src/main.rs:70 on main

     0.000203401s  INFO examples_otlp: try to call `curl -i http://127.0.0.1:3003/` (with trace)
    at src/main.rs:71 on main

     0.000213920s  INFO examples_otlp: try to call `curl -i http://127.0.0.1:3003/heatlh` (with NO trace)
    at src/main.rs:72 on main
...

Into an other terminal, call the / (endpoint with opentelemetry_tracing_layer and response_with_trace_layer)

❯ curl -i http://127.0.0.1:3003/
HTTP/1.1 200 OK
content-type: application/json
content-length: 50
traceparent: 00-b2611246a58fd7ea623d2264c5a1e226-b2c9b811f2f424af-01
tracestate:
date: Wed, 28 Dec 2022 17:04:59 GMT

{"my_trace_id":"b2611246a58fd7ea623d2264c5a1e226"}

call the /health (endpoint with NO layer)

❯ curl -i http://127.0.0.1:3003/health
HTTP/1.1 200 OK
content-type: application/json
content-length: 15
date: Wed, 28 Dec 2022 17:14:07 GMT

{"status":"UP"}

Compatibility

axum axum-tracing-opentelemetry
0.6 latest - 0.6
0.5 0.1 - 0.5

History

0.8

  • add init_propagator to configure the global propagator based on content of the env variable OTEL_PROPAGATORS

  • add a layerresponse_with_trace_layer to have traceparent injected into response

  • improve discovery of otlp configuration based on OTEL_EXPORTER_OTLP_TRACES_ENDPOINT, OTEL_EXPORTER_OTLP_ENDPOINT, OTEL_EXPORTER_OTLP_TRACES_PROTOCOL, OTEL_EXPORTER_OTLP_PROTOCOL

0.6

  • upgrade to axum 0.6

0.5

  • upgrade to opentelemetry 0.18
  • breaking change: upgrade opentelemetry-jaeger to 0.17 (switch from PipelineBuiler to AgentPipeline)

0.4

  • allow customization of tracer
  • add tracer to export on stdout or stderr
  • add tracer to export to nowhere (like /dev/null) to allow to have trace_id and the opentelemetry span & metadata on log and http response (without collector)

0.3

  • Allow customization of exporter pipeline
  • Fix name of the root span (#6)

0.2

  • First public release as a crate

0.1