trace-tools

Tracing and diagnostic tools for async, using tokio and tracing.

The crate consists of a few components:

• The Observe trait
• A tracing::Subscriber implementation, as well as useful Layers.
• Diagnostic utilities (currently just one, in Flamegrapher).
• #[observe] attribute procedural macro.

The Observe trait

This trait provides the ability to wrap futures with an instrumentation span, similar to the tracing_futures::Instrument trait. This span has some default properties that make it easy to register and filter in a subscriber:

tracing::trace_span!(
    target: "trace_tools::observe",
    "observed",
    observed.name = <my_function_name>,
    loc.file = <caller_file>,
    loc.line = <caller_line>,
    loc.col = <caller_column>,
);

Crucially, this span contains identical fields to those generated internally by tokio for task spawns (when compiled with --cfg tokio_unstable, which means that subscribers can interact with spans generated by Observe in the same way that they can with tokios internal spans.

Subscriber implementation

trace-tools provides a subscriber and two Layers for dealing with spans and events:

1. Flamegraph layer: produces a folded stack file detailing instrumented span stacks (either instrumented internally by tokio or by trace-tools) that can be used to generate a flamegraph of all observed code.
2. Logging layer: allows log records to be converted into tracing events, and recreates full logging functionality (equivalent to fern-logger) within the subscriber itself, since it is impossible to set two loggers/subscribers at once. This means that logging remains consistent whether using trace-tools or not.
3. Console layer (enabled by the tokio-console feature): builds a console_subscriber::ConsoleLayer (from tokio's console project) to collect task metrics and broadcast them. With this layer enabled, you can run the console binary and observe all asynchronous tasks in real-time. Note: only spans associated with tasks are observed in this way, not all spans; trace_tools::observe spans will not appear.

The subscriber can be initialised through a builder that can either set the global subscriber or return a Layered instance that can be further extended with more Layers.

Setting as the default subscriber:

// `Flamegrapher` handle returned, for building a flamegraph at the end of the run.
let flamegrapher = trace_tools::subscriber::build()
    .with_flamegraph_layer(stack_filename)
    .with_log_layer(log_config)
    .init()?
    .unwrap();

Returning a Layered struct:

let (subscriber, flamegrapher) = trace_tools::subscriber::build()
    .with_flamegraph_layer(stack_filename)
    .with_log_layer(log_config)
    .finish()?;

// Extend the subscriber with external layers.
let subscriber = subscriber.with(console_layer);

// Set the global subscriber.
subscriber.init();

Flamegrapher

Produces a flamegraph using the given folded stack file, using the inferno crate.

let flamegrapher = trace_tools::subscriber::build()
    .with_flamegraph_layer(stack_filename)
    .init()?
    .unwrap();

// Run some instrumented code...
// ...
// ...

// Programatically create the flamegraph file.
flamegrapher
    .with_graph_file("flamegraph.svg")?
    .write_flamegraph()?;

#[observe] attribute

trace-tools provides a simple attribute proc-macro for instrumenting functions and futures with a span that specifies the trace_tools::observe target. The location fields in the span will describe the location of the tagged function or future, and the observed.name field will specify the function name:

use trace_tools::observe;

#[observe]
pub async fn say_hello() {
    println!("hello");
}

This is equivalent to the following:

trace_tools::Observe(say_hello(), "say_hello").await;

This macro can be used with regular, non-async functions too, unlike the Observe trait.

tokio-console feature

The tokio-console feature enables the console layer. Note that this makes use of unstable tokio features in order to work. As such, this also crate must be built with RUSTFLAGS="--cfg tokio_unstable" to use the feature.

Examples

There is an example for each layer in trace-tools/examples. The flamegraph example produces this interactive graph: