Crate tracing_tunnel

Expand description

Tunnelling tracing information across an API boundary.

This crate provides tracing infrastructure helpers allowing to transfer tracing events across an API boundary:

TracingEventSender is a tracing Subscriber that converts tracing events into (de)serializable presentation that can be sent elsewhere using a customizable hook.
TracingEventReceiver consumes events produced by a TracingEventSender and relays them to the tracing infrastructure. It is assumed that the source of events may outlive both the lifetime of a particular TracingEventReceiver instance, and the lifetime of the program encapsulating the receiver. To deal with this, the receiver provides the means to persist / restore its state.

When is this needed?

This crate solves the problem of having dynamic call sites for tracing spans / events, i.e., ones not known during compilation. This may occur if call sites are defined in dynamically loaded modules, the execution of which is embedded into the program, e.g., WASM modules.

It could be feasible to treat such a module as a separate program and collect / analyze its traces in conjunction with host traces using distributed tracing software (e.g., OpenTelemetry / Jaeger). However, this would significantly bloat the API surface of the module, bloat its dependency tree, and would arguably break encapsulation.

The approach proposed in this crate keeps the module API as simple as possible: essentially, a single function to smuggle TracingEvents through the client–host boundary. The client side (i.e., the TracingEventSender) is almost stateless; it just streams tracing events to the host, which can have tracing logic as complex as required.

Another problem that this crate solves is having module executions that can outlive the host program. For example, WASM module instances can be fully persisted and resumed later, potentially after the host is restarted. To solve this, TracingEventReceiver allows persisting call site data and alive spans, and resuming from the previously saved state (notifying the tracing infra about call sites / spans if necessary).

Use case: workflow automation

Both components are used by the Tardigrade workflows, in case of which the API boundary is the WASM client–host boundary.

The tardigrade client library uses TracingEventSender to send tracing events from a workflow (i.e., a WASM module instance) to the host using a WASM import function.
The Tardigrade runtime uses TracingEventReceiver to pass traces from the workflow to the host tracing infrastructure.

Crate features

Each of the two major features outlined above is gated by the corresponding opt-in feature, sender and receiver. Without these features enabled, the crate only provides data types to capture tracing data.

`std`

(On by default)

Enables support of types from std, such as the Error trait. Propagates to tracing-core, enabling Error support there.

Even if this feature is off, the crate requires the global allocator (i.e., the alloc crate) and u32 atomics.

`sender`

(Off by default)

Provides TracingEventSender.

`receiver`

(Off by default; requires std)

Provides TracingEventReceiver and related types.

Examples

Sending events with `TracingEventSender`

use tracing_tunnel::{TracingEvent, TracingEventSender, TracingEventReceiver};

// Let's collect tracing events using an MPSC channel.
let (events_sx, events_rx) = mpsc::sync_channel(10);
let subscriber = TracingEventSender::new(move |event| {
    events_sx.send(event).ok();
});

tracing::subscriber::with_default(subscriber, || {
    tracing::info_span!("test", num = 42_i64).in_scope(|| {
        tracing::warn!("I feel disturbance in the Force...");
    });
});

let events: Vec<_> = events_rx.iter().collect();
assert!(!events.is_empty());
// There should be one "new span".
let span_count = events
    .iter()
    .filter(|event| matches!(event, TracingEvent::NewSpan { .. }))
    .count();
assert_eq!(span_count, 1);

Receiving events from `TracingEventReceiver`

tracing_subscriber::fmt().pretty().init();

let events: Vec<TracingEvent> = // ...

let mut spans = PersistedSpans::default();
let mut local_spans = LocalSpans::default();
// Replay `events` using the default subscriber.
let mut receiver = TracingEventReceiver::default();
for event in events {
    if let Err(err) = receiver.try_receive(event) {
        tracing::warn!(%err, "received invalid tracing event");
    }
}
// Persist the resulting receiver state. There are two pieces
// of the state: metadata and alive spans.
let metadata = receiver.persist_metadata();
let (spans, local_spans) = receiver.persist();
// `metadata` can be shared among multiple executions of the same executable
// (e.g., a WASM module).
// `spans` and `local_spans` are specific to the execution; `spans` should
// be persisted, while `local_spans` should be stored in RAM.

Structs

CallSiteData

Data for a single tracing call site: either a span definition, or an event definition.

DebugObject

Opaque wrapper for a Debuggable object recorded as a value in a tracing span or event.

LocalSpansreceiver

Subscriber-specific information about tracing spans for a particular execution (e.g., a WASM module instance).

PersistedMetadatareceiver

Information about span / event Metadata that is serializable and thus can be persisted across multiple TracingEventReceiver lifetimes.

PersistedSpansreceiver

Information about alive tracing spans for a particular execution that is (de)serializable and can be persisted across multiple TracingEventReceiver lifetimes.

TracedErrorstd

(De)serializable presentation for an error recorded as a value in a tracing span or event.

TracedValues

Collection of named TracedValues.

TracedValuesIter

Iterator over name-value references returned from TracedValues::iter().

TracingEventReceiverreceiver

Receiver of TracingEvents produced by TracingEventSender that relays them to the tracing infrastructure.

TracingEventSendersender

Tracing Subscriber that converts tracing events into (de)serializable presentation that can be sent elsewhere using a customizable hook.