rialo-telemetry 0.3.0

# rialo-telemetry

A comprehensive telemetry library for distributed tracing and metrics in Rialo applications. This crate provides a unified interface for setting up OpenTelemetry distributed tracing, Prometheus metrics, and console logging with minimal configuration.

## Features

- **OpenTelemetry Integration**: Full support for distributed tracing with OTLP HTTP exporters
- **OpenTelemetry Metrics**: Configuration support for OTLP metrics export (implementation pending)
- **Distributed Tracing**: Centralized utilities for HTTP trace context propagation
- **Baggage Support**: Complete baggage manipulation utilities for distributed metadata propagation
- **Prometheus Metrics**: Optional span latency metrics and custom registry support
- **Console Logging**: Configurable structured logging to console
- **Environment Variable Configuration**: Automatic configuration from standard OpenTelemetry environment variables
- **Flexible Configuration**: Builder pattern for programmatic configuration
- **Feature Gated**: Optional dependencies based on your needs

## Optional Features

- `axum-headers` - Enables HTTP server trace context extraction utilities for axum
- `distributed-tracing` - Enables OpenTelemetry distributed tracing support
- `env-context` - Enables environment variable-based trace context propagation for subprocess communication
- `prometheus` - Enables Prometheus metrics collection
- `reqwest-headers` - Enables HTTP client trace context injection utilities for reqwest

## Quick Start

### Console-Only Logging

```rust
use rialo_telemetry::{TelemetryConfig, init_telemetry};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    // Initialize with console logging only
    let config = TelemetryConfig::new();
    let handle = init_telemetry(config).await?;
    
    // Your application code here
    tracing::info!("Application started");
    
    // Shutdown telemetry when done
    handle.shutdown()?;
    Ok(())
}
```

### OpenTelemetry with Environment Variables

Enable the `distributed-tracing` feature and set environment variables:

```bash
export OTEL_SERVICE_NAME="my-service"
export OTEL_SERVICE_VERSION="1.0.0"
export OTEL_EXPORTER_OTLP_ENDPOINT="http://localhost:4318"
export OTEL_EXPORTER_OTLP_HEADERS="x-api-key=your-key"
```

```rust
use rialo_telemetry::{TelemetryConfig, init_telemetry};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    // Initialize with OTLP support (reads from environment)
    let config = TelemetryConfig::new().with_otlp();
    let handle = init_telemetry(config).await?;
    
    // Your traced application code here
    tracing::info!("Application started with distributed tracing");
    
    handle.shutdown()?;
    Ok(())
}
```

### Programmatic Configuration

```rust
use rialo_telemetry::{TelemetryConfig, OtlpConfig, init_telemetry};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let otlp_config = OtlpConfig::new()
        .with_service_name("my-service")
        .with_service_version("1.0.0")
        .with_exporter_endpoint("https://api.honeycomb.io/v1/traces")
        .with_console_enabled(true);
    
    let config = TelemetryConfig::new()
        .with_otlp_config(otlp_config)
        .with_log_level("debug");
    
    let handle = init_telemetry(config).await?;
    
    // Your application code
    
    handle.shutdown()?;
    Ok(())
}
```

### OpenTelemetry Metrics Configuration

The crate includes full configuration support for OpenTelemetry metrics export, though the actual metrics implementation is not yet active. All environment variables and configuration options are parsed and stored for future use:

```rust
use rialo_telemetry::{TelemetryConfig, OtlpConfig, init_telemetry};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let otlp_config = OtlpConfig::new()
        .with_service_name("my-service")
        // Traces endpoint
        .with_traces_endpoint("http://jaeger:14268/api/traces")
        // Metrics configuration (ready for future implementation)
        .with_exporter_endpoint("http://otel-collector:4318"); // Base endpoint for metrics
    
    let config = TelemetryConfig::new()
        .with_otlp_config(otlp_config);
    
    let handle = init_telemetry(config).await?;
    
    // When metrics are implemented, they will automatically use the configured endpoints
    
    handle.shutdown()?;
    Ok(())
}
```

**Note**: While metrics configuration is fully supported, the actual metrics export implementation is planned for a future release. Currently, only tracing is actively exported via OTLP.

### Prometheus Metrics

Enable the `prometheus` feature:

```rust
use rialo_telemetry::{TelemetryConfig, PrometheusConfig, init_telemetry};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let registry = prometheus::Registry::new();
    
    let prometheus_config = PrometheusConfig::new(registry.clone())
        .with_span_latency_buckets(20)
        .with_span_latency_enabled(true);
    
    let config = TelemetryConfig::new()
        .with_prometheus_config(prometheus_config);
    
    let handle = init_telemetry(config).await?;
    
    // Your application code - span latencies will be recorded
    
    handle.shutdown()?;
    Ok(())
}
```

## Distributed Tracing Context Propagation

The crate provides utilities for propagating trace context across HTTP requests, enabling distributed tracing across microservices.

### HTTP Client (reqwest) - Trace Context Injection

Enable the `reqwest-headers` feature to inject trace context into outgoing HTTP requests:

```rust
use rialo_telemetry::{inject_trace_headers, apply_trace_headers_to_reqwest};
use reqwest::Client;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    // Initialize telemetry with OpenTelemetry
    let config = TelemetryConfig::new().with_otlp();
    let handle = init_telemetry(config).await?;
    
    let client = Client::new();
    
    // In a traced context
    let span = tracing::info_span!("http_request", service = "api-call");
    let _guard = span.enter();
    
    // Inject trace headers into request
    let trace_headers = inject_trace_headers();
    let request = client.post("https://api.example.com/data");
    let request = apply_trace_headers_to_reqwest(request, trace_headers);
    
    let response = request.send().await?;
    
    handle.shutdown()?;
    Ok(())
}
```

### HTTP Server (axum) - Trace Context Extraction

Enable the `axum-headers` feature to extract trace context from incoming HTTP requests:

```rust
use rialo_telemetry::extract_and_set_trace_context_axum;
use axum::{extract::State, http::HeaderMap, Json, response::Json as ResponseJson};

#[tracing::instrument]
async fn handler(
    headers: HeaderMap,
    Json(payload): Json<serde_json::Value>
) -> ResponseJson<serde_json::Value> {
    // Extract and set trace context from incoming headers
    extract_and_set_trace_context_axum(&headers);
    
    // Your handler logic - this span will now be part of the distributed trace
    tracing::info!("Processing request with distributed trace context");
    
    ResponseJson(serde_json::json!({"status": "ok"}))
}
```

### End-to-End Distributed Tracing Example

Combining both client and server utilities for full distributed tracing:

```rust
// Service A (HTTP client)
use rialo_telemetry::{TelemetryConfig, init_telemetry, inject_trace_headers, apply_trace_headers_to_reqwest};

async fn call_service_b() -> Result<(), Box<dyn std::error::Error>> {
    let span = tracing::info_span!("call_service_b");
    let _guard = span.enter();
    
    let client = reqwest::Client::new();
    let trace_headers = inject_trace_headers();
    
    let request = client.post("http://service-b:8080/api/process");
    let request = apply_trace_headers_to_reqwest(request, trace_headers);
    
    let response = request.send().await?;
    tracing::info!("Received response from service B");
    
    Ok(())
}

// Service B (HTTP server) 
use rialo_telemetry::extract_and_set_trace_context_axum;
use axum::{http::HeaderMap, Json};

#[tracing::instrument]
async fn process_request(
    headers: HeaderMap,
    Json(data): Json<serde_json::Value>
) -> Json<serde_json::Value> {
    // Extract trace context - this creates a child span of service A's span
    extract_and_set_trace_context_axum(&headers);
    
    tracing::info!("Processing request in service B");
    
    // This span is now part of the same distributed trace as service A
    let result = process_business_logic(data).await;
    
    Json(result)
}
```

**Note**: Both utilities require the `distributed-tracing` feature to be enabled along with their respective feature flags (`reqwest-headers` or `axum-headers`).

### Environment Variable Context Propagation

Enable the `env-context` feature to propagate trace context across process boundaries using environment variables.

#### Trace Context Inheritance

When the `env-context` feature is enabled, you can manually extract trace context from environment variables after initializing telemetry. This allows subprocesses to connect to their parent's trace:

```rust
use rialo_telemetry::{TelemetryConfig, init_telemetry, extract_and_set_trace_context_env};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    // Initialize telemetry first
    let config = TelemetryConfig::new().with_otlp();
    let handle = init_telemetry(config).await?;
    
    // Then extract trace context from environment variables (if any)
    extract_and_set_trace_context_env();
    
    // This span is now part of parent's distributed trace!
    tracing::info!("Child process started");
    
    handle.shutdown()?;
    Ok(())
}
```

#### Ergonomic Command Helper

Use `inject_trace_env_to_cmd()` for convenient one-liner subprocess trace propagation:

```rust
use rialo_telemetry::{TelemetryConfig, init_telemetry, inject_trace_env_to_cmd};
use std::process::Command;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    // Initialize telemetry first
    let config = TelemetryConfig::new().with_otlp();
    let handle = init_telemetry(config).await?;
    
    // Then extract trace context from environment variables (if any)
    extract_and_set_trace_context_env();
    
    // Parent process: ergonomic one-liner to inject trace context
    let span = tracing::info_span!("subprocess_execution", command = "worker");
    let _guard = span.enter();
    
    // One line replaces the manual env injection loop!
    let cmd = inject_trace_env_to_cmd(Command::new("./worker"));
    let output = cmd.arg("--task=process").output()?;
    
    tracing::info!("Subprocess completed with status: {}", output.status);
    
    handle.shutdown()?;
    Ok(())
}
```

#### Manual Control (Advanced Usage)

For fine-grained control, you can still use the manual functions:

```rust
use rialo_telemetry::{inject_trace_env, extract_and_set_trace_context_env};
use std::process::Command;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    // Initialize telemetry first
    let config = TelemetryConfig::new().with_otlp();
    let handle = init_telemetry(config).await?;
    
    // Manually extract trace context at a specific point
    extract_and_set_trace_context_env();
    
    // Manual injection (equivalent to inject_trace_env_to_cmd)
    let trace_env = inject_trace_env();
    let mut cmd = Command::new("./child-process");
    for (key, value) in trace_env {
        cmd.env(key, value);
    }
    
    let output = cmd.output()?;
    handle.shutdown()?;
    Ok(())
}
```

In the child process, initialize telemetry and extract trace context:

```rust
use rialo_telemetry::{TelemetryConfig, init_telemetry, extract_and_set_trace_context_env};

#[tracing::instrument]
fn main() -> Result<(), Box<dyn std::error::Error>> {
    let rt = tokio::runtime::Runtime::new()?;
    rt.block_on(async {
        // Initialize telemetry first
        let config = TelemetryConfig::new().with_otlp();
        let handle = init_telemetry(config).await?;
        
        // Then extract trace context from environment variables
        extract_and_set_trace_context_env();
        
        // This span is now part of the parent process's distributed trace
        tracing::info!("Child process started with inherited trace context");
        
        do_work().await;
        tracing::info!("Child process completed");
        
        handle.shutdown()
    })
}

async fn do_work() {
    let span = tracing::info_span!("child_work");
    let _guard = span.enter();
    
    tracing::info!("Performing work in child process");
    // This work is automatically traced as part of the parent's trace
}
```

You can also extract from a custom environment map instead of the current process environment:

```rust
use rialo_telemetry::extract_and_set_trace_context_from_env_map;
use std::collections::HashMap;

fn handle_custom_environment(custom_env: &HashMap<String, String>) {
    // Extract trace context from a specific environment map
    extract_and_set_trace_context_from_env_map(custom_env);
    
    // Current span now has the extracted trace context as parent
    tracing::info!("Working with custom trace context");
}
```

### Cross-Process Distributed Tracing Example

Complete example showing trace propagation from parent to child process:

```rust
// parent.rs - The main process
use rialo_telemetry::{TelemetryConfig, init_telemetry, inject_trace_env_to_cmd};
use std::process::Command;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let config = TelemetryConfig::new().with_otlp();
    let handle = init_telemetry(config).await?;
    
    // Start a distributed trace
    let span = tracing::info_span!("batch_job", job_id = "12345");
    let _guard = span.enter();
    
    tracing::info!("Starting batch job with multiple workers");
    
    // Launch multiple child processes with ergonomic one-liners
    for worker_id in 1..=3 {
        let worker_span = tracing::info_span!("launch_worker", worker_id = worker_id);
        let _worker_guard = worker_span.enter();
        
        // Ergonomic one-liner - inject trace context and spawn
        let cmd = inject_trace_env_to_cmd(Command::new("./worker"))
            .arg(worker_id.to_string());
        
        tracing::info!("Launching worker {}", worker_id);
        cmd.spawn()?;
    }
    
    tracing::info!("All workers launched");
    
    handle.shutdown()?;
    Ok(())
}

// worker.rs - Child process
use rialo_telemetry::{TelemetryConfig, init_telemetry, extract_and_set_trace_context_env};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let worker_id = std::env::args().nth(1).unwrap_or("unknown".to_string());
    
    // Initialize telemetry first
    let config = TelemetryConfig::new().with_otlp();
    let handle = init_telemetry(config).await?;
    
    // Then extract trace context from environment variables
    extract_and_set_trace_context_env();
    
    // Create main span for this worker - now inherits from parent
    let worker_span = tracing::info_span!("worker_process", worker_id = worker_id);
    let _guard = worker_span.enter();
    
    tracing::info!("Worker {} started with inherited trace", worker_id);
    
    // Do work - all automatically part of the original batch job trace
    process_batch_items().await;
    
    tracing::info!("Worker {} completed", worker_id);
    
    handle.shutdown()?;
    Ok(())
}

async fn process_batch_items() {
    for item in 1..=10 {
        let item_span = tracing::info_span!("process_item", item_id = item);
        let _guard = item_span.enter();
        
        tracing::info!("Processing item {}", item);
        // Simulate work
        tokio::time::sleep(std::time::Duration::from_millis(100)).await;
    }
}
```

**Note**: Environment variable context propagation requires the `distributed-tracing` feature to be enabled along with the `env-context` feature flag.

## Baggage Support

Baggage provides a way to propagate key-value metadata across distributed systems alongside trace context. It's useful for passing cross-cutting concerns like user IDs, feature flags, request priorities, or any other data that should be available throughout a distributed trace.

The crate provides comprehensive baggage manipulation utilities when the `distributed-tracing` feature is enabled:

### Basic Baggage Operations

```rust
use rialo_telemetry::{get_baggage, get_all_baggage};
use rialo_telemetry::{TelemetryConfig, init_telemetry};
use opentelemetry::{baggage::{Baggage, BaggageExt, BaggageMetadata}, Context, Key, Value};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    // Initialize telemetry with OpenTelemetry (baggage requires this)
    let config = TelemetryConfig::new().with_otlp();
    let handle = init_telemetry(config).await?;

    // To set baggage, create a context with baggage and attach it within a scope
    let span = tracing::info_span!("my_operation");
    let _guard = span.enter();

    // Create baggage and attach to context
    {
        let mut baggage = Baggage::new();
        baggage.insert_with_metadata(
            Key::new("user_id".to_string()),
            Value::from("12345".to_string()),
            BaggageMetadata::default(),
        );
        let context = Context::current().with_baggage(baggage);
        let _baggage_guard = context.attach();

        // Baggage is now available within this scope
        let user_id = get_baggage("user_id"); // Some("12345")
        let request_id = get_baggage("request_id"); // None

        // Get all baggage as HashMap
        let all_baggage = get_all_baggage();
        println!("Current baggage: {:?}", all_baggage);
    }

    handle.shutdown()?;
    Ok(())
}
```

### Distributed Baggage Propagation

Baggage automatically propagates across distributed systems through the same mechanisms as trace context:

```rust
use rialo_telemetry::{get_baggage, inject_trace_headers, apply_trace_headers_to_reqwest};
use reqwest::Client;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let config = TelemetryConfig::new().with_otlp();
    let handle = init_telemetry(config).await?;

    let span = tracing::info_span!("call_service_b");
    let _guard = span.enter();

    // HTTP call - baggage automatically included in trace context headers if present
    let client = Client::new();
    let trace_headers = inject_trace_headers(); // Includes any baggage from current context
    let request = client.post("http://service-b:8080/process");
    let request = apply_trace_headers_to_reqwest(request, trace_headers);

    let response = request.send().await?;

    handle.shutdown()?;
    Ok(())
}

// Service B: Automatically receives baggage
use rialo_telemetry::{extract_and_set_trace_context_axum, get_baggage};
use axum::{http::HeaderMap, Json};

#[tracing::instrument]
async fn process_request(
    headers: HeaderMap,
    Json(data): Json<serde_json::Value>
) -> Json<serde_json::Value> {
    // Extract trace context (includes baggage)
    extract_and_set_trace_context_axum(&headers);
    
    // Baggage is now available in service B
    let user_id = get_baggage("user_id"); // Some("user-12345")
    let tenant_id = get_baggage("tenant_id"); // Some("tenant-abc")
    let priority = get_baggage("request_priority"); // Some("high")
    
    tracing::info!("Processing request for user {:?} in tenant {:?} with priority {:?}",
                  user_id, tenant_id, priority);
    
    // Business logic can use baggage data
    if priority == Some("high".to_string()) {
        process_with_high_priority(data).await
    } else {
        process_normally(data).await
    }
}
```

### Cross-Process Baggage Propagation

Baggage also propagates across process boundaries when using environment variable context propagation:

```rust
use rialo_telemetry::inject_trace_env_to_cmd;
use std::process::Command;
use opentelemetry::baggage::BaggageExt;
use opentelemetry::Context;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let config = TelemetryConfig::new().with_otlp();
    let handle = init_telemetry(config).await?;

    // Parent process: Set baggage using OpenTelemetry context
    let baggage = opentelemetry::baggage::Baggage::builder()
        .with_entry("batch_id", "batch-2024-001")
        .with_entry("processing_mode", "parallel")
        .build();

    let cx = Context::current().with_baggage(baggage);
    let _guard = cx.attach();

    let span = tracing::info_span!("launch_worker");
    let _span_guard = span.enter();

    // Launch subprocess with baggage propagation
    let cmd = inject_trace_env_to_cmd(Command::new("./worker"));
    let output = cmd.arg("--task=process").output()?;

    handle.shutdown()?;
    Ok(())
}

// Worker process: Inherits baggage
use rialo_telemetry::{TelemetryConfig, init_telemetry, extract_and_set_trace_context_env, get_baggage};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    // Initialize telemetry first
    let config = TelemetryConfig::new().with_otlp();
    let handle = init_telemetry(config).await?;

    // Extract trace context and baggage from environment variables
    extract_and_set_trace_context_env();

    // Baggage from parent is now available
    let batch_id = get_baggage("batch_id"); // Some("batch-2024-001")
    let mode = get_baggage("processing_mode"); // Some("parallel")

    tracing::info!("Worker started for batch {:?} in mode {:?}", batch_id, mode);

    // Use baggage to influence processing
    if mode == Some("parallel".to_string()) {
        process_in_parallel().await;
    } else {
        process_sequentially().await;
    }

    handle.shutdown()?;
    Ok(())
}
```

### Baggage Best Practices

**Use Cases:**
- User identification across services
- Feature flags and A/B testing
- Request prioritization and routing
- Tenant or organization context
- Debug flags and trace sampling decisions

**Performance Considerations:**
- Keep baggage small (recommended: < 1KB total)
- Use short, meaningful keys
- Create new contexts with updated baggage when values need to change
- Drop context guards when exiting scopes to restore previous context

**Security Notes:**
- Don't put sensitive data in baggage (it's propagated in headers)
- Baggage is visible to all services in the trace
- Consider baggage as public metadata within your distributed system

### Baggage Configuration

Baggage propagation is enabled by default when OpenTelemetry is configured. The propagators include both trace context and baggage:

```rust
// Baggage is included in default propagators
// OTEL_PROPAGATORS="tracecontext,baggage" (default)

// To customize propagators (not recommended unless you have specific needs):
std::env::set_var("OTEL_PROPAGATORS", "tracecontext,baggage,b3");
```

**Note**: Baggage utilities require the `distributed-tracing` feature to be enabled.

## Environment Variables

The crate supports all standard OpenTelemetry environment variables:

### Service Configuration
- `OTEL_SERVICE_NAME` - Service name (default: "rialo")
- `OTEL_SERVICE_VERSION` - Service version (default: "unknown")
- `OTEL_RESOURCE_ATTRIBUTES` - Additional resource attributes

### Endpoint Configuration
- `OTEL_EXPORTER_OTLP_ENDPOINT` - General OTLP endpoint (default: "http://localhost:4318")
- `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT` - Traces-specific endpoint (overrides general)
- `OTEL_EXPORTER_OTLP_METRICS_ENDPOINT` - Metrics-specific endpoint (overrides general)
- `OTEL_EXPORTER_OTLP_INSECURE` - Use insecure connection for general endpoint (default: false)
- `OTEL_EXPORTER_OTLP_TRACES_INSECURE` - Use insecure connection for traces (default: false)
- `OTEL_EXPORTER_OTLP_METRICS_INSECURE` - Use insecure connection for metrics (default: false)

### Headers
- `OTEL_EXPORTER_OTLP_HEADERS` - General headers for authentication
- `OTEL_EXPORTER_OTLP_TRACES_HEADERS` - Traces-specific headers (merged with general)
- `OTEL_EXPORTER_OTLP_METRICS_HEADERS` - Metrics-specific headers (merged with general)

### Protocol Configuration
- `OTEL_EXPORTER_OTLP_PROTOCOL` - General export protocol: "grpc", "http/protobuf", "http/json" (default: "http/protobuf")
- `OTEL_EXPORTER_OTLP_TRACES_PROTOCOL` - Traces-specific protocol (overrides general)
- `OTEL_EXPORTER_OTLP_METRICS_PROTOCOL` - Metrics-specific protocol (overrides general)

### Feature Toggles
- `OTEL_TRACES_ENABLED` - Enable/disable traces (default: true)
- `OTEL_METRICS_ENABLED` - Enable/disable metrics (default: true)
- `OTEL_LOG_LEVEL` - Log level (default: "info")

### Metrics Configuration
- `OTEL_EXPORTER_OTLP_METRICS_PERIOD` - Metrics reporting interval (default: "30s")

### Propagation
- `OTEL_PROPAGATORS` - Trace context propagators, comma-separated (default: "tracecontext,baggage")

## Local Jaeger Setup

For local development and testing, you can easily connect to a local Jaeger instance to visualize your traces.

### Running Jaeger

Jaeger can be run either as an all-in-one Docker container or built locally. A Nix recipe is included for convenience in the [rialo-nix-toolchain](https://github.com/SubzeroLabs/rialo-nix-toolchain).

#### Docker (Recommended for Quick Testing)

Run the all-in-one Jaeger container:

```bash
docker run -d --name jaeger \
  -e COLLECTOR_ZIPKIN_HTTP_PORT=9411 \
  -p 5775:5775/udp \
  -p 6831:6831/udp \
  -p 6832:6832/udp \
  -p 5778:5778 \
  -p 16686:16686 \
  -p 14268:14268 \
  -p 9411:9411 \
  jaegertracing/all-in-one:1.6.0
```

This exposes the following ports:
- **16686**: Jaeger UI (http://localhost:16686)
- **14268**: Jaeger collector (HTTP)
- **4318**: OTLP gRPC endpoint (if using OTLP collector)

#### Nix (For Development Environment)

If you're using the rialo-nix-toolchain, you can run Jaeger with:

```bash
nix run .#jaeger
```

### Connecting Your Application

Once Jaeger is running, configure your application to send traces to it using environment variables:

```bash
export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318
export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=http://localhost:4318/v1/traces
export OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
```

Or create a `.env` file:

```bash
# .env
OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318
OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=http://localhost:4318/v1/traces
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
```

### Viewing Traces

1. Start your application with the environment variables set
2. Generate some traces by using your application
3. Open http://localhost:16686 in your browser
4. Select your service from the dropdown
5. Click "Find Traces" to see your traces

## Configuration Precedence

Configuration values are resolved in this order (highest to lowest precedence):
1. Programmatic configuration via builder methods
2. Environment variables
3. Default values

For endpoints and headers, signal-specific settings override general settings:
- `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT` overrides `OTEL_EXPORTER_OTLP_ENDPOINT` for traces
- `OTEL_EXPORTER_OTLP_METRICS_ENDPOINT` overrides `OTEL_EXPORTER_OTLP_ENDPOINT` for metrics
- Signal-specific headers are merged with general headers (signal-specific takes precedence for conflicts)

## Examples

### Honeycomb.io Integration

```bash
export OTEL_SERVICE_NAME="my-service"
export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT="https://api.honeycomb.io/v1/traces"
export OTEL_EXPORTER_OTLP_HEADERS="x-honeycomb-team=your-api-key,x-honeycomb-dataset=my-dataset"

# If using metrics (when implemented):
# export OTEL_EXPORTER_OTLP_METRICS_ENDPOINT="https://api.honeycomb.io/v1/metrics"
```

### Jaeger Integration

```bash
export OTEL_SERVICE_NAME="my-service"
export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT="http://localhost:14268/api/traces"
export OTEL_EXPORTER_OTLP_PROTOCOL="http/protobuf"

# Jaeger doesn't support metrics via OTLP, so disable them:
# export OTEL_METRICS_ENABLED="false"
```

### Separate Traces and Metrics Endpoints

```bash
export OTEL_SERVICE_NAME="my-service"
# Send traces to Jaeger
export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT="http://jaeger:14268/api/traces"
# Send metrics to Prometheus-compatible endpoint (when implemented)
export OTEL_EXPORTER_OTLP_METRICS_ENDPOINT="http://prometheus:9090/api/v1/otlp/v1/metrics"
# Different authentication for each
export OTEL_EXPORTER_OTLP_TRACES_HEADERS="authorization=Bearer traces-token"
export OTEL_EXPORTER_OTLP_METRICS_HEADERS="authorization=Bearer metrics-token"
```

### Development Setup

```bash
# Console logging only for development
export RUST_LOG="debug"
# No OTEL_EXPORTER_OTLP_ENDPOINT set - will use console only
```

## Error Handling

The library handles common error scenarios gracefully:

- **Invalid endpoints**: Empty or invalid endpoints disable OpenTelemetry export
- **Network failures**: Export failures don't crash the application
- **Configuration errors**: Invalid environment variables fall back to defaults
- **Global subscriber conflicts**: Handles multiple initialization attempts gracefully

## Performance Considerations

- **Batched Export**: Uses OpenTelemetry's batched span processor for efficient export
- **Conditional Compilation**: Feature gates ensure zero overhead when features are disabled
- **Efficient Headers**: Headers are parsed once and reused
- **Resource Detection**: Uses OpenTelemetry's resource detection for optimal metadata

## Testing

```bash
# Test with no features
cargo nextest run -p rialo-telemetry

# Test with Prometheus
cargo nextest run -p rialo-telemetry --features prometheus
```

## License

Licensed under the Apache License, Version 2.0.

## Contributing

This crate is part of the Rialo project. See the main repository for contribution guidelines.