Sideways OTel 🦀

Observability from the side - because crabs walk sideways, and so should your telemetry.

A production-ready Rust telemetry library that provides vendor-neutral OpenTelemetry tracing, metrics, and logs, exported via OTLP to any compatible backend - a local Collector, the .NET Aspire dashboard, or any hosted vendor that speaks OTLP.

Note: This crate was built with substantial AI assistance (Claude Code) - design, implementation, and documentation were all done in collaboration with an AI agent, with human review and testing against real OTLP backends throughout. We're noting this in the interest of transparency.

Features

🔭 Vendor-Neutral OTLP Export - Traces, metrics, and logs exported over gRPC to any OTLP-compatible backend
🚀 One-Line Initialization - Simple init_telemetry() call sets up everything
🔧 Standard OTEL_* Env Vars - Configure via the environment variables already defined by the OpenTelemetry spec
💪 Graceful Degradation - Continues running even if the OTLP endpoint is unavailable
📈 Native OpenTelemetry Metrics - No vendor-specific macros; sync and observable (async/callback) instruments come straight from the OTel metrics API
🧵 Span Helpers - Small, generic helpers for attaching attributes and events to the current span
🔍 Health Check Filtering - Automatically filters out noisy health check spans

Quick Start

Add to your Cargo.toml:

[dependencies]
sideways-otel = "0.2"

Initialize in your application:

use sideways_otel::{init_telemetry, TelemetryConfig};
use sideways_otel::prelude::*;
use tracing::info;

fn main() {
    // Load from environment variables
    let config = TelemetryConfig::from_env();
    let telemetry = init_telemetry(&config);

    // Use tracing as normal
    info!("Application started!");

    // Emit metrics using the native OpenTelemetry API
    let requests = counter("requests.handled");
    requests.add(1, &[KeyValue::new("status", "success")]);

    // ... your application code ...

    // Flush and shut down on exit (important!)
    telemetry.shutdown();
}

Configuration

Environment Variables

TelemetryConfig::from_env() reads the standard OpenTelemetry environment variables, plus a few additions for enabling/disabling individual signals:

# Service identity
OTEL_SERVICE_NAME=my-service
OTEL_RESOURCE_ATTRIBUTES=deployment.environment=production,team=platform

# OTLP export
OTEL_EXPORTER_OTLP_PROTOCOL=grpc              # "grpc" (default) or "http/protobuf"
OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317   # 4317 for grpc, 4318 for http/protobuf
OTEL_EXPORTER_OTLP_HEADERS=x-api-key=some-secret,x-tenant=acme

# Enable/disable individual signals (default: true for all)
OTEL_TRACES_ENABLED=true
OTEL_METRICS_ENABLED=true
OTEL_LOGS_ENABLED=true

# Metrics export interval, in milliseconds (default: 60000)
OTEL_METRIC_EXPORT_INTERVAL=60000

# Logging level - full tracing_subscriber directive syntax is supported,
# including per-target overrides and "off":
RUST_LOG=warn,h2=off,hyper=off,tonic=off,opentelemetry=off,opentelemetry_sdk=off

# Enable JSON-formatted console logging (default: false)
JSON_LOGGING=false

OTEL_EXPORTER_OTLP_HEADERS is the mechanism for passing whatever a given backend needs for authentication (an API key, a tenant header, etc.) - the library itself has no knowledge of any particular vendor. See examples/vendor_backend.rs for a worked, vendor-agnostic example (endpoint and header both come from environment variables, so it works unmodified against whichever OTLP-compatible backend you point it at).

Both http:// and https:// endpoints work for either protocol - https:// automatically gets a TLS-enabled channel trusting Mozilla's bundled root store (via tonic's tls-webpki-roots), so no extra configuration is needed to reach a public vendor endpoint.

OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf is useful where gRPC/HTTP2 gets blocked (some corporate proxies/gateways) or where a vendor only exposes an HTTP ingestion endpoint. When left unset, the exporter falls back to its protocol-specific default endpoint (http://localhost:4317 for gRPC, http://localhost:4318 for HTTP/protobuf) if OTEL_EXPORTER_OTLP_ENDPOINT also isn't set.

Programmatic Configuration

use sideways_otel::TelemetryConfig;

let config = TelemetryConfig::builder()
    .service_name("my-service")
    .otlp_endpoint("http://localhost:4317")
    .with_otlp_header("x-api-key", "some-secret")
    .with_resource_attribute("deployment.environment", "production")
    .build();

let telemetry = sideways_otel::init_telemetry(&config);

Metrics

Metrics are created and recorded through the native OpenTelemetry metrics API - there are no macros to import. sideways_otel::prelude provides counter/histogram/up_down_counter/gauge shortcuts, each scoped automatically to the service name passed to init_telemetry - no need to repeat it at every call site:

use sideways_otel::prelude::*;

let requests = counter("requests.handled");
requests.add(1, &[KeyValue::new("status", "success"), KeyValue::new("endpoint", "/users")]);

let latency = histogram("request.duration_ms");
latency.record(42.5, &[KeyValue::new("endpoint", "/users")]);

let queue_depth = up_down_counter("queue.depth");
queue_depth.add(1, &[KeyValue::new("queue", "emails")]);

These default to u64 counters, f64 histograms/gauges, and i64 up/down counters - the common case. For a different numeric type, per-instrument description/unit, or a meter scoped to something other than the service name, drop down to meter() (or opentelemetry::global::meter("some-scope")) and its u64_counter()/f64_histogram()/etc. builders directly.

Create each instrument once (e.g. in a OnceLock or at startup) and reuse it - meter() is cheap to call repeatedly, but recreating the instrument itself on every call is wasted work.

Observable (Async) Metrics

For a value that's cheaper or more natural to sample on demand than to push on every change - a queue depth, open file descriptors, current pool size - use the observable variants instead. Rather than calling .add()/.record() yourself, you register a callback once and the SDK invokes it on every collection cycle:

use sideways_otel::prelude::*;

// The returned handle can be dropped immediately - the callback keeps
// firing for the life of the MeterProvider regardless, since registration
// happens against the SDK's meter pipeline, not the handle itself.
let _open_connections = observable_gauge("db.pool.open_connections", |observer| {
    observer.observe(current_pool_size() as f64, &[KeyValue::new("pool", "primary")]);
});

let _queue_depth = observable_up_down_counter("queue.depth", |observer| {
    observer.observe(current_queue_depth(), &[KeyValue::new("queue", "emails")]);
});

let _requests_total = observable_counter("requests.total", |observer| {
    observer.observe(current_request_count(), &[]);
});
# fn current_pool_size() -> i64 { 0 }
# fn current_queue_depth() -> i64 { 0 }
# fn current_request_count() -> u64 { 0 }

A callback can call observer.observe(...) more than once with different attributes, to report several related values (e.g. per-queue depths) from a single registration. There's no observable_histogram - the OTel spec doesn't define one, since a histogram's whole point is recording a distribution of individual measurements as they happen, not a single sampled value.

Tracing

Once telemetry is initialized, use the standard tracing crate for distributed tracing.

Basic Logging

use tracing::{info, warn, error};

info!("Application started");
warn!(user_id = 123, "Rate limit approaching");
error!(error = ?err, "Failed to process request");

Instrumentation

#[tracing::instrument] wraps a function in a span - by default named after the function, with every argument recorded as a field (via Debug), and child of whatever span was active when it was called:

#[tracing::instrument]
async fn process_request(id: u64) {
    tracing::info!(request_id = id, "Processing request");
    // ... do work ...
}

Adding More Data to the Span

The attribute takes several options for going beyond the defaults - useful when the arguments alone don't tell the whole story, contain things you don't want recorded, or when a value is only known partway through the function:

#[tracing::instrument(
    name = "orders.process",        // override the span name (default: the function name)
    skip(order),                    // don't try to Debug-format `order` as a field
    fields(
        order.id = %order.id,       // %  -> format with Display
        order.total_cents = order.total_cents,
        order.status = tracing::field::Empty, // declared now, filled in once known
    ),
    err,                             // if this returns Err, record it as a field automatically
)]
async fn process_order(order: &Order) -> Result<(), OrderError> {
    charge_card(&order.id).await?;

    // Record a field that wasn't known when the span was created.
    tracing::Span::current().record("order.status", "shipped");

    Ok(())
}

A few things worth knowing:

skip(arg1, arg2, ...) (or skip_all) excludes arguments from the span's fields - required for anything that isn't Debug, and good practice for large payloads or secrets you don't want in your telemetry backend.
fields(...) adds fields beyond the function's arguments. Prefix a value with % to format it with Display, ? for Debug (the default for argument-derived fields), or leave it bare for values that already implement tracing::Value (integers, bools, strings).
tracing::field::Empty reserves a field slot for a value you don't have yet; call tracing::Span::current().record("field_name", &value) later in the function once you do. Fields not declared up front (in fields(...) or the argument list) can't be recorded this way.
err (or err(Debug) to use Debug instead of Display) automatically records an Err return value as a field and emits an event for it - handy instead of manually calling record_error at every fallible call site.
level = "debug" (or trace/warn/error) controls the span's level; default is info.

Span Attribute Helpers

sideways_otel::span (re-exported from the prelude) provides generic helpers for enriching the current span with OpenTelemetry attributes and events, without reaching for tracing_opentelemetry directly:

use sideways_otel::prelude::*;

#[tracing::instrument]
async fn process_order(order_id: &str) {
    set_attribute(KeyValue::new("order.id", order_id.to_string()));

    if let Err(err) = charge_card(order_id).await {
        record_error(&err);
    }

    add_event("order.completed", [KeyValue::new("order.id", order_id.to_string())]);
}

Health Check Filtering

The library automatically filters out health check-related spans to reduce noise:

Spans from tonic_health
Spans containing "health", "Health", or "Check"
gRPC health check services

Local Testing

Any OTLP-compatible collector works. Two easy options:

OpenTelemetry Collector (Docker):

docker run -p 4317:4317 -p 4318:4318 otel/opentelemetry-collector:latest

.NET Aspire dashboard (standalone, no .NET project required) exposes an OTLP/gRPC endpoint and a browser UI for traces, logs, and metrics:

docker run --rm -it -p 18888:18888 -p 18889:18889 \
  mcr.microsoft.com/dotnet/aspire-dashboard:latest

Then point your service at it:

OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:18889 OTEL_SERVICE_NAME=my-service cargo run

Open http://localhost:18888 (using the browser auth token printed to the container's console) to see traces, structured logs, and metrics.

Publishing

This crate is published to crates.io.

Before publishing:

cargo test --all-features
cargo clippy --all-targets --all-features

Then:

cargo publish --dry-run   # Verify the package contents and metadata
cargo publish             # Actually publish

cargo publish requires you to be logged in (cargo login) with an account that has publish rights on the crate, and will refuse to publish if Cargo.toml metadata (description, license, readme, etc.) is incomplete.

License

Dual-licensed under either of

at your option.

sideways-otel 0.2.1