# tracing-calltree
Use `tracing-calltree` when you want an always-on hierarchical performance view of a Rust application already instrumented with `tracing`.
## What it does
`tracing-calltree` observes `tracing` spans and builds a persistent semantic call tree from their ancestry.
Each tree node maintains rolling statistics over recent completed span invocations, including:
* minimum duration
* maximum duration
* mean duration
* p95 duration
* retained sample count
* lifetime call count
It records three timing views:
* **wall** — elapsed time from first span entry until span close
* **active** — accumulated time while the span is entered
* **suspended** — `wall - active`
Active time is **not CPU time**.
The default rolling window is 100 completed invocations per tree node.
## Mental model
Given instrumented operations:
```rust
#[tracing::instrument(skip_all)]
async fn handle_request() {
load_context().await;
}
#[tracing::instrument(skip_all)]
async fn load_context() {
query_database().await;
}
#[tracing::instrument(skip_all)]
async fn query_database() {
// ...
}
```
the profiler can produce a tree resembling:
```text
handle_request
└── load_context
└── query_database
```
with recent timing statistics attached to every node.
The hierarchy comes from normal `tracing` span ancestry. Do not implement a separate call stack or task-local hierarchy when using this crate.
## Basic setup
```rust
use tracing_calltree::CallTree;
use tracing_subscriber::prelude::*;
let calltree = CallTree::builder()
.window_size(100)
.build();
tracing_subscriber::registry()
.with(calltree.layer())
.init();
```
The profiler can coexist with other `tracing-subscriber` layers:
```rust
tracing_subscriber::registry()
.with(tracing_subscriber::fmt::layer())
.with(calltree.layer())
.init();
```
## Reading the profiler
Take an owned snapshot:
```rust
let snapshot = calltree.snapshot();
```
For human-readable diagnostics:
```rust
println!("{}", snapshot.display());
```
Snapshots are independent of the profiler's internal synchronization and are intended to be consumed by application-defined diagnostics, CLIs, TUIs, APIs, exporters, tests, or other tooling.
The crate itself intentionally does not provide an HTTP server, collector, or external runtime service.
## Async instrumentation
Prefer normal `tracing` async instrumentation.
Whole functions:
```rust
#[tracing::instrument(skip_all)]
async fn lookup() {
// ...
}
```
Local futures:
```rust
use tracing::Instrument;
async {
lookup().await;
}
.instrument(tracing::info_span!("lookup"))
.await;
```
Do **not** hold a synchronous `Span::enter()` guard across `.await`.
An instrumented async future may enter and exit its span multiple times as the future is polled. `tracing-calltree` accounts for this and records one completed sample when the span closes.
## Interpreting wall vs active time
A node such as:
```text
database_query
wall p95: 180 ms
active p95: 8 ms
suspended p95: 172 ms
```
suggests that most of the operation's latency occurs while it is not actively entered, such as while waiting on I/O, scheduling, locks, timers, or other asynchronous work.
By contrast:
```text
parse_document
wall p95: 42 ms
active p95: 40 ms
```
suggests that most of the elapsed duration occurs while the span is actively executing.
Do not interpret active duration as processor usage.
## Tree semantics
The tree is based on semantic tracing spans, not every Rust function call.
A child with the same name under different parents represents separate tree nodes:
```text
request
└── database
maintenance
└── database
```
Unprofiled or otherwise irrelevant tracing spans may occur between profiled operations without necessarily needing to appear in the resulting semantic tree.
Avoid high-cardinality span structures. Dynamic values such as user IDs, request IDs, filenames, or database keys should normally be span **fields**, not dynamically generated span names.
## Filtering
A production application often contains many tracing spans that are useful for logs but inappropriate for continuous profiling.
Use normal `tracing-subscriber` layer filtering or the library's filtering facilities to restrict profiling to meaningful semantic operations.
Good profiling spans generally represent operations such as:
```text
handle_request
load_context
database_query
acquire_connection
serialize_response
execute_job
```
Avoid creating timing-tree nodes for very small implementation details unless they are diagnostically useful.
## Rolling-window semantics
Latency statistics describe only the retained recent sample window.
For example, with a window of 100:
```text
total_calls = 25,000
samples = 100
```
means the node has executed 25,000 times, while its min/max/mean/p95 describe only its most recent 100 retained completions.
This makes the profiler useful for answering:
> What has been slow recently?
rather than only reporting lifetime distributions.
## Serde
Enable the optional `serde` feature when snapshots need to be serialized:
```toml
tracing-calltree = { version = "0.1", features = ["serde"] }
```
Keep serialization, HTTP APIs, persistence, and external monitoring systems outside the core profiling model.
## When to use tracing-calltree
Use it when:
* the application already uses `tracing`
* semantic parent/child operation structure matters
* recent behavior matters more than long-term high-resolution distributions
* the application should expose or inspect performance data while it continues running
* an external profiler should not be required for routine diagnostics
It is particularly suitable for long-running services and daemons.
## Development guidance
When modifying code that uses `tracing-calltree`:
1. Preserve ordinary `tracing` span ancestry rather than manually constructing call-tree paths.
2. Prefer static, meaningful span names and avoid dynamic cardinality.
3. Use `#[tracing::instrument(skip_all)]` for whole async functions when appropriate.
4. Use `.instrument(span)` for local async scopes.
5. Never hold `Span::enter()` across `.await`.
6. Treat snapshot data as recent diagnostic state rather than globally atomic metrics.
7. Remember that wall, active, and suspended statistics have different semantics.
8. Avoid flattening the hierarchy prematurely when presenting diagnostics.
9. Keep application-specific transport and presentation concerns outside the core profiler.
10. Measure instrumentation overhead before introducing additional fine-grained spans.
## Quick diagnostic workflow
When investigating a slow application:
```text
snapshot
↓
find root with high wall p95
↓
inspect its children
↓
follow the expensive branch
↓
compare wall vs active
↓
identify likely execution-heavy or wait-heavy operation
```
The key advantage of `tracing-calltree` is that this diagnostic hierarchy is continuously maintained from ordinary `tracing` instrumentation rather than requiring a separate profiling session.