Crate stackpulse

Expand description

A library for recording Linux CPU stack samples into a compact spool file and, when you want them, resolving the recorded addresses into displayable frames. Built to drop into profilers, capture agents, benchmark harnesses, and developer tooling.

The crate covers two phases: capture and replay. Recording always runs; symbolization is opt-in. Embedders that already have their own symbol pipeline can read raw instruction pointers straight from the spool and never touch PerfSymbolizer. stackpulse does not aggregate stacks, render flame graphs, or pick an output format. Those decisions stay with the caller.

§Quick example

use std::time::{Duration, Instant};
use stackpulse::{AttachMode, PerfRecorder, PerfRecorderOptions, PerfSpoolReader, PerfSymbolizer};
let mut recorder = PerfRecorder::attach(
    pid,
    "profile.spool",
    AttachMode::StopAttachEnableResume,
    PerfRecorderOptions { frequency: 99, stack_size: 60 * 1024, ..Default::default() },
)?;

let deadline = Instant::now() + Duration::from_secs(10);
while Instant::now() < deadline && recorder.process_is_active(pid as i32) {
    recorder.wait()?;
    recorder.consume_available()?;
}
recorder.finish()?;

let reader = PerfSpoolReader::open("profile.spool")?;
let mut symbolizer = PerfSymbolizer::for_spool(&reader);

for stack in reader.sample_stacks() {
    symbolizer.for_each_sample_stack(stack, |frame| {
        println!("{}", frame.func_name());
    });
}

§Core types

Type	Role
`PerfRecorder`	Attaches to one or more processes, drains `perf_event_open` ring buffers, writes a spool file.
`PerfSpoolReader`	Reads a spool file back into samples, modules, exec markers, interned stack frames, and borrowed frame contexts.
`PerfSymbolizer`	Resolves raw frame addresses using ELF symbols, kernel symbols, Python perf maps, and address fallbacks. The native ELF backend is pluggable via `NativeSymbolizer`.
`NativeSymbolizer`	Trait for swapping in your own native symbolizer (custom debuginfod, debug-dir, or source-info policy). `PerfSymbolizer` still handles kernel and perf-map frames.
`profile` types	Resolved frame data types: what an aggregator, UI, or exporter consumes.

Recording and symbolization are deliberately separate. The recorder writes a self-contained spool file; symbolization happens later, off the hot path, and can run on a different host as long as the binaries and perf maps are preserved.

§Raw replay

For integrations with an existing symbolizer, consume raw frames directly:

let reader = stackpulse::PerfSpoolReader::open("profile.spool")?;

for sample in reader.samples() {
    for context in reader.stack_frame_contexts(sample.process_id, sample.stack_id)? {
        let ip = context.frame.abs_ip;
        if let Some(module) = context.module {
            // Pass `ip`, `module.module`, and `module.rel_ip` to your symbolizer.
        }
    }
}

stack_frame_contexts does not symbolize. It only binds borrowed raw frames to the module mapping stackpulse recorded at capture time.

§Plugging in an external native symbolizer

Callers with their own debuginfod, debug-dir, or source-info pipeline can keep using PerfSymbolizer for kernel and perf-map frames while substituting a different backend for native ELF modules. Implement NativeSymbolizer and hand a factory to PerfSymbolizer::with_native_factory (or PerfSymbolizer::for_spool_with_native_factory). stackpulse parses each module’s ELF, computes its image base, and calls set_modules whenever the module set for a process group changes; you then receive symbolize_one(addr) for every native frame:

use std::rc::Rc;
use stackpulse::{
    NativeSymbolizer, NativeSymbolizerFactory, PerfSpoolReader, PerfSymbolizer,
    SymModule, SymbolsRc,
};

struct MySymbolizer { /* your wholesym / debuginfod / dwarf state */ }

impl NativeSymbolizer for MySymbolizer {
    fn set_modules(&mut self, modules: Vec<SymModule>) {
        // modules carries path, avma_range, and ModuleImageBase already
        // resolved from ELF. No /proc or /maps work needed here.
    }
    fn symbolize_one(&mut self, addr: u64) -> SymbolsRc {
        // Convert addr -> SVMA via the SymModule's image_base, then look up.
        Rc::from([])
    }
}

let reader = PerfSpoolReader::open("profile.spool")?;
let factory: NativeSymbolizerFactory = Box::new(|_pid: i32| -> Box<dyn NativeSymbolizer> {
    Box::new(MySymbolizer { /* ... */ })
});
let mut symbolizer = PerfSymbolizer::for_spool_with_native_factory(&reader, true, factory);

Kernel frames (/proc/kallsyms) and Python or JIT perf maps (/tmp/perf-PID.map) stay inside PerfSymbolizer; the plug-in only sees native module addresses. The default factory (default_native_symbolizer_factory) returns the bundled wholesym backend and is what the no-argument constructors install.

§Vocabulary

A sample is one timestamped observation of one thread.
A module is an executable memory range: a binary, shared object, anonymous JIT mapping, or kernel range.
A raw frame is an address recorded in the spool file.
A resolved frame is a displayable ResolvedFrame produced by PerfSymbolizer.
A spool file is the compact on-disk profile written by PerfRecorder.

§Runtime requirements

Linux only. Uses perf_event_open, /proc, ELF metadata, optional /proc/kallsyms, and optional Python perf maps under /tmp.

User-space recording works as the same user that owns the target. Kernel frames, containers, hardened systems, and aggressive sample rates may need extra capabilities (typically CAP_PERFMON) or a relaxed perf_event_paranoid setting. See the Permissions section in the explanation chapter for the full breakdown.

§Tutorials

§Attach to an existing process

Pick a CPU-bound target. Python emits perf-map entries for its own frames when run with perf support, so the resolved output shows function names instead of bare addresses:

PYTHONPERFSUPPORT=1 python3 -X perf - <<'PY'
import os
print(os.getpid(), flush=True)
v = 0
while True:
    v = (v * 33 + 17) % 1000003
PY

Attach to that PID, drain for ten seconds, then read the spool file back:

use std::time::{Duration, Instant};
use stackpulse::{AttachMode, PerfRecorder, PerfRecorderOptions, PerfSpoolReader, PerfSymbolizer};

fn record(pid: u32) -> stackpulse::Result<()> {
    let mut recorder = PerfRecorder::attach(
        pid,
        "profile.spool",
        AttachMode::StopAttachEnableResume,
        PerfRecorderOptions { frequency: 99, stack_size: 60 * 1024, ..Default::default() },
    )?;

    let deadline = Instant::now() + Duration::from_secs(10);
    while Instant::now() < deadline && recorder.process_is_active(pid as i32) {
        recorder.wait()?;
        recorder.consume_available()?;
    }
    recorder.finish()?;

    let reader = PerfSpoolReader::open("profile.spool")?;
    let mut symbolizer = PerfSymbolizer::for_spool(&reader);

    for stack in reader.sample_stacks().take(10) {
        println!("pid={} tid={}", stack.sample.process_id, stack.sample.thread_id);
        symbolizer.for_each_sample_stack(stack, |f| {
            println!("  {}", f.func_name());
        });
    }
    Ok(())
}

The wait/consume_available pair is the recording loop. wait blocks until a ring buffer is readable; consume_available drains the queued records, unwinds the samples it finds, and writes them to the spool. If you skip the pair the kernel buffers fill up and subsequent samples are dropped, showing up as lost_events in the summary.

Samples reference stack IDs, not inline frame data, which is why profiles stay small when hot code keeps producing the same stacks. It is also why you should reuse a single PerfSymbolizer: it caches resolved frames keyed by (process_id, stack_id).

§Capture process startup

Attaching to a running process misses early startup. To profile from the first instruction, launch the child suspended, attach with AttachWithEnableOnExec, and let it run:

use std::ffi::{OsStr, OsString};
use std::time::{Duration, Instant};
use stackpulse::{process::SuspendedLaunchedProcess, AttachMode, PerfRecorder, PerfRecorderOptions};
let args = [OsString::from("-X"), OsString::from("perf"), OsString::from("-c"),
    OsString::from("v = 0\nfor _ in range(50_000_000):\n    v = (v + 1) % 1009\n")];
let env = [(OsString::from("PYTHONPERFSUPPORT"), OsString::from("1"))];

let launched = SuspendedLaunchedProcess::launch_in_suspended_state(
    OsStr::new("python3"), &args, &env,
)?;

let mut recorder = PerfRecorder::attach(
    launched.pid(),
    "startup.spool",
    AttachMode::AttachWithEnableOnExec,
    PerfRecorderOptions { frequency: 199, stack_size: 60 * 1024, ..Default::default() },
)?;

let running = launched.unsuspend_and_run()?;
let timeout = Instant::now() + Duration::from_secs(30);

let status = loop {
    if let Some(status) = running.try_wait()? { break status; }
    if Instant::now() >= timeout {
        recorder.disable();
        return Err("child did not exit before timeout".into());
    }
    recorder.wait()?;
    recorder.consume_available()?;
};

recorder.consume_available()?;
let summary = recorder.finish()?;
println!("status={status:?} samples={}", summary.samples);

The kernel enables the perf events on execve, so nothing is recorded before the child has loaded its binary, and nothing is missed once it starts running.

§Aggregate into a stack histogram

Printing each frame is a debugging mode. A real exporter counts how often each stack appears. This snippet is the kernel of any flame-graph or top-functions report:

use std::collections::BTreeMap;
use stackpulse::{PerfSpoolReader, PerfSymbolizer};
let reader = PerfSpoolReader::open("profile.spool")?;
let mut symbolizer = PerfSymbolizer::for_spool(&reader);
let mut counts = BTreeMap::<String, u64>::new();

for stack in reader.sample_stacks() {
    let mut names = Vec::new();
    symbolizer.for_each_sample_stack(stack, |f| {
        names.push(f.func_name());
    });
    let key = names.join(";");
    *counts.entry(key).or_default() += 1;
}

let mut rows: Vec<_> = counts.into_iter().collect();
rows.sort_by(|a, b| b.1.cmp(&a.1).then_with(|| a.0.cmp(&b.0)));
for (stack, count) in rows.iter().take(20) {
    println!("{count:>8} {stack}");
}

A production exporter keeps more metadata around: process and thread IDs, timestamps, FrameKind, SymbolOrigin, file names, and line numbers. Most exporters also hide frames flagged with FrameFlags::HIDDEN_DEFAULT in their default view.

§Recipes

Self-contained snippets for the recurring tweaks you make once the basic record-then-symbolize loop is up: choosing recording options, attaching to a running process, replaying a spool you saved earlier, swapping in a custom symbolizer, and handling errors that come back from the recorder.

§Pick recording options

Start conservative:

use stackpulse::PerfRecorderOptions;

let options = PerfRecorderOptions {
    frequency: 99,
    stack_size: 60 * 1024,
    include_kernel: false,
    inherit_child_processes: false,
    ..Default::default()
};

Knobs:

Field	When to change it	What it costs
`frequency`	Need more or fewer samples per second.	Higher rates raise CPU overhead and increase the chance of lost events under load.
`stack_size`	Stacks are getting truncated.	More memory copied per sample. Capped at `MAX_SAMPLE_USER_STACK`.
`include_kernel`	Want syscall, scheduler, or kernel-lock attribution.	Usually needs extra privileges. If only kernel sampling is denied, `PerfRecorder::attach` retries user-only and reports `summary.kernel_enabled = false`.
`inherit_child_processes`	Forked children are part of the workload.	Opens more perf events and adds bookkeeping per child.
`start_timestamp_us`	Aligning the profile to an external clock or trace.	Metadata only; read back through `PerfSpoolReader::timestamp_us`.
`sample_interval_us`	UI or export format wants an interval hint.	Metadata only; does not drive kernel sampling.

Check the kernel cap before asking for an aggressive rate:

if let Some(limit) = stackpulse::max_sample_rate() {
    println!("kernel cap: {limit}");
}

§Drain without dropping samples

Call consume_available inside the loop and once more before finish:

while recorder.process_is_active(pid as i32) {
    recorder.wait()?;
    recorder.consume_available()?;
}
recorder.consume_available()?;
let summary = recorder.finish()?;

Draining is mandatory. A recorder that you only hold onto will not write anything to the spool: wait parks the thread until perf data arrives, and consume_available is what turns that data into spool records. If you already have an event loop, run wait from a worker thread or poll PerfRecorder::has_pending_events from the main loop and drain when it returns true.

§Profile more than one process

After attaching the first PID, add the others:

use stackpulse::AttachMode;
recorder.open_process(other_pid, AttachMode::StopAttachEnableResume)?;

To pick up everything under a known root:

for child in stackpulse::children::discover_all_descendants(root_pid) {
    recorder.open_process(child as u32, AttachMode::StopAttachEnableResume)?;
}

§Follow children created after recording starts

Turn on inherit_child_processes:

let options = stackpulse::PerfRecorderOptions {
    inherit_child_processes: true,
    ..Default::default()
};

The recorder watches for forks, clones the parent’s module state, and opens the new process. Children that existed before recording started aren’t picked up automatically; attach them yourself.

§Catch threads created later

Perf inheritance usually catches new threads. When it doesn’t (or you’ve deliberately turned it off to limit fan-out), refresh periodically:

recorder.refresh_threads(pid)?;

This scans /proc/<pid>/task and opens events for threads it hasn’t seen. Run it from a slow maintenance tick, not the hot loop.

§Resolve symbols

One symbolizer per profile, reused for every sample:

let reader = stackpulse::PerfSpoolReader::open("profile.spool")?;
let mut symbolizer = stackpulse::PerfSymbolizer::for_spool(&reader);

for stack in reader.sample_stacks() {
    symbolizer.for_each_sample_stack(stack, |frame| {
        // render or aggregate
        let _ = frame.func_name();
    });
}

for_each_sample_stack streams borrowed frames straight out of the symbolizer’s cache. It only stores compact frame ids per repeated (process_id, stack_id), so callers that render or aggregate inline never pay for materializing a full resolved-stack Vec.

Display policy is your call. Most UIs hide FrameFlags::HIDDEN_DEFAULT by default, group frames by FrameKind, and surface SymbolOrigin in detail views so users can tell ELF symbols from address-only fallbacks.

§Use your own symbolizer

If your application already owns symbolization, skip PerfSymbolizer and read raw frames plus their recorded module context directly:

let reader = stackpulse::PerfSpoolReader::open("profile.spool")?;

for sample in reader.samples() {
    for context in reader.stack_frame_contexts(sample.process_id, sample.stack_id)? {
        let frame = context.frame;
        let module = context.module;
        // Resolve `frame.abs_ip` using your own native, JIT, or kernel symbolizer.
        // `module` is only the recorded mapping context.
    }
}

Use sample_stacks() when you only need raw frames and sample metadata:

for sample_stack in reader.sample_stacks() {
    for frame in sample_stack.frames {
        let _ = frame.abs_ip;
    }
}

This is the path to use when you have your own debug-dir, debuginfod, Python perf-map, JIT, or kernel-symbol pipeline and want stackpulse to stay out of the way.

§Python frames

PerfSymbolizer reads Python perf maps when the runtime emits them. For modern CPython:

PYTHONPERFSUPPORT=1 python3 -X perf app.py

The default symbolizer allows perf-map lookup for any PID:

let mut symbolizer = stackpulse::PerfSymbolizer::new(reader.modules());

The spool file only stores Python runtime markers, not the perf-map content itself. If you want to symbolize later (on another machine, or after the runtime has cleaned up /tmp/perf-<pid>.map), copy those maps next to the spool.

To avoid stale perf maps from PID reuse, restrict lookup to processes the recorder last saw as Python runtimes:

let mut python_pids = std::collections::BTreeSet::new();
for exec in reader.process_execs() {
    if exec.is_python_runtime {
        python_pids.insert(exec.process_id);
    } else {
        python_pids.remove(&exec.process_id);
    }
}

let mut symbolizer = stackpulse::PerfSymbolizer::with_perf_map_processes(
    reader.modules(), python_pids,
);

PerfSymbolizer::for_spool_with_recorded_python_perf_maps(reader) is a broader convenience helper: it allows any PID that was ever marked as a Python runtime in the spool.

Or skip perf maps entirely:

let mut symbolizer = stackpulse::PerfSymbolizer::with_perf_maps(reader.modules(), false);

§Kernel frames

Set include_kernel:

let options = stackpulse::PerfRecorderOptions {
    include_kernel: true,
    ..Default::default()
};

Kernel frames come from perf callchains. User frames still go through the native DWARF unwinder; the user side of the perf callchain is only consulted when DWARF unwinding stops early or returns nothing. Anything from the user callchain that the DWARF result already covered is counted as ignored_user_callchain_frames in the summary.

After attach, check whether kernel sampling actually stuck:

let summary = recorder.summary();
if !summary.kernel_enabled {
    eprintln!("fell back to user-only frames");
}

Kernel names come from /proc/kallsyms when readable; otherwise kernel frames render as addresses.

§Diagnose bad profiles

PerfSummary is the first place to look:

let summary = recorder.finish()?;
println!("events: {}",   summary.sample_events);
println!("written: {}",  summary.samples);
println!("lost: {}",     summary.lost_events);
println!("empty: {}",    summary.empty_stack_samples);
println!("truncated: {}",summary.truncated_frame_markers);
println!("errors: {}",   summary.error_stats.total());

Reading the numbers:

Symptom	Likely cause	Fix
`sample_events > samples`	Samples lacked PIDs, TIDs, timestamps, or frames.	Look at the specific skip counters.
High `lost_events`	Ring buffers overran.	Lower `frequency`, drain more often, reduce fan-out.
High `empty_stack_samples`	Register/stack capture failed, or unwind produced nothing.	Check `summary.error_stats`.
Lots of truncation	`stack_size` too small.	Bump it, up to `MAX_SAMPLE_USER_STACK`.
Mostly address-only frames	No symbols or mappings available.	Keep the binaries; symbolize on a host that has them.

For a formatted breakdown:

let mut report = String::new();
stackpulse::ErrorStatsFormatter::new(
    &summary.error_stats, summary.sample_events, summary.samples,
).write_to(&mut report)?;
println!("{report}");

§Permission failures

Permission errors surface when opening perf events, reading /proc, asking for kernel frames, or reading /proc/kallsyms. Work down the list:

Profile a process you own.
Drop include_kernel.
Cap your request at or below stackpulse::max_sample_rate().
Grant CAP_PERFMON (or whatever your kernel requires) to the profiler binary.
Relax perf_event_paranoid in test environments.

Don’t fail the whole recording on the first permission error. If kernel sampling alone was denied, PerfRecorder::attach has already retried in user-only mode and surfaced that through summary.kernel_enabled.

§Reference

A condensed map of the public surface. Each item links to its full rustdoc page.

§Module map

The crate root re-exports the recording, reading, and symbolization types:

use stackpulse::{
    AttachMode, PerfRecorder, PerfRecorderOptions, PerfSpoolReader, PerfSymbolizer,
};

Public modules:

Module	What it’s for
`process`	Launch a process suspended before `execve` so sampling starts at birth.
`children`	Walk descendant PIDs through `/proc`.
`profile`	Resolved frames and symbol data types.
`state`	Process liveness, exit watching, and signal helpers.

§Recording

§`PerfRecorder`

Records stack samples for one or more processes and writes a spool file.

Method	What it does
`attach(pid, output, mode, options)`	Open perf events, create the spool, register known mappings, start sampling.
`consume_available()`	Drain perf data, update module/process state, unwind, write records.
`wait()`	Block briefly for new perf data.
`open_process(pid, mode)`	Add another process to the same recording.
`refresh_threads(pid)`	Discover new threads when perf inheritance isn’t doing it.
`disable()`	Stop sampling for all attached events.
`has_pending_events()`	Is there perf data ready to drain?
`summary()`	Snapshot of recording counters.
`process_is_active(pid)`	Is a given PID still alive?
`has_active_processes_except(pid)`	Is any PID other than the given one still alive?
`finish()`	Flush, return final counters, consume the recorder.

The recorder is not just a handle. Opening one and never calling consume_available will fill kernel buffers and lose samples.

§`AttachMode`

Variant	Use
`StopAttachEnableResume`	Attaching to a running process. The target is briefly stopped while events open, then resumed.
`AttachWithEnableOnExec`	Attaching to a forked-but-not-yet-exec’d child. Pair with `process::SuspendedLaunchedProcess`.

§`PerfRecorderOptions`

Field	Type	Meaning
`frequency`	`u32`	Samples per second. Must be ≤ `/proc/sys/kernel/perf_event_max_sample_rate` when readable.
`stack_size`	`u32`	User stack bytes copied per sample. Capped at `MAX_SAMPLE_USER_STACK`.
`include_kernel`	`bool`	Capture kernel frames when allowed.
`inherit_child_processes`	`bool`	Follow children forked after recording starts.
`start_timestamp_us`	`u64`	Timeline anchor stored in the spool.
`sample_interval_us`	`u64`	Optional interval hint stored in the spool.

Default zero-fills everything. For a real recording, set frequency and stack_size at minimum.

§`PerfSummary`

Counter snapshot for quality checks.

Field	Meaning
`sample_events`	Raw perf sample records seen.
`samples`	Samples written to the spool.
`lost_events`	Kernel-reported losses.
`kernel_enabled`	Whether kernel capture stayed on after attach.
`missing_pid_samples` / `missing_tid_samples`	Samples dropped for missing IDs.
`idle_tid_samples`	Samples attributed to idle TID 0.
`missing_timestamp_samples`	Samples without a perf timestamp.
`empty_stack_samples`	Samples that produced no usable frames.
`truncated_frame_markers`	Unwind truncation markers observed.
`ignored_user_callchain_frames`	User frame-pointer callchain frames dropped because DWARF unwinding supplied those frames.
`error_stats`	Per-kind sample error counters.

§Reading spool files

§`PerfSpoolReader`

PerfSpoolReader::open(path) reads the whole spool into memory and validates record references.

Method	What it returns
`start_timestamp_us()`	Profile timeline anchor stored in the spool header.
`sample_interval_us()`	Optional sample interval metadata stored in the spool header.
`modules()`	Recorded executable memory ranges.
`frames()`	Interned raw frame records. Useful for precomputing symbolization caches.
`samples()`	Timestamped samples.
`process_execs()`	Process exec markers, including Python runtime on/off.
`recovered_from_truncated_tail()`	Whether the spool ended mid-record and the reader kept only the intact prefix.
`kernel_frame_addresses()`	Iterator over absolute kernel IPs in interned frames. Used by `PerfSymbolizer::for_spool` for sparse `kallsyms` loading.
`stack_frame_refs(stack_id)`	Borrow raw `FrameRecord`s for an interned stack without copying.
`stack_frame_contexts(pid, stack_id)`	Borrow raw frames with recorded module context for an interned stack.
`sample_stacks()`	Iterate samples with borrowed raw stacks.
`stack_frames(stack_id, out)`	Expand an interned stack into `FrameRecord`s. Clears `out` first.
`timestamp_us(sample)`	Sample timestamp in profile-timeline microseconds.

Frame iteration order is leaf to root. FrameModuleRef::rel_ip uses the same file-offset coordinate space as FrameRecord::rel_ip; external symbolizers can combine it with the recorded module mapping however their own lookup API requires.

§`ModuleRecord`

Field	Meaning
`id`	Stable module ID within this profile.
`process_id`	Owning PID (or a kernel marker for kernel code).
`start`, `end`	Runtime address range.
`file_offset`	File offset matching `start`.
`inode`	Backing file inode, when known.
`path`	Path or display name as `ModulePath`. Spool-read paths can borrow from the mmap-backed profile.
`is_kernel`	Kernel range?

§`FrameRecord`

Field	Meaning
`module_id`	Matched module, when known.
`rel_ip`	Module-relative address.
`abs_ip`	Absolute IP.
`mode`	`FrameMode::User`, `FrameMode::Kernel`, or `FrameMode::TruncatedStackMarker`.

FrameRecord::truncated_stack_marker() creates the sentinel written when native unwinding stopped before the stack root. Use FrameRecord::is_truncated_stack_marker() to detect it in raw-frame workflows.

§`OwnedSampleRecord`

Field	Meaning
`timestamp_ns`	Monotonic perf timestamp (ns).
`process_id`	PID.
`thread_id`	TID.
`stack_id`	Pass to `PerfSpoolReader::stack_frames`.

§`ProcessExecRecord`

Field	Meaning
`timestamp_ns`	Monotonic timestamp (ns).
`process_id`	PID.
`is_python_runtime`	Latest observation: does this PID look like a Python runtime with perf-map support? A later marker with `false` means stop treating it as Python.

§Symbolization

§`PerfSymbolizer`

Resolves raw frames into displayable ones. One per profile, reused.

Constructor or method	Use
`new(modules)`	Default: ELF, kernel symbols, plus Python perf maps for any PID.
`for_spool(reader)`	Create a symbolizer for a loaded spool, including sparse kernel-symbol loading.
`for_spool_with_perf_maps(reader, allow)`	Same as `for_spool`, but explicitly enable or disable Python perf maps.
`for_spool_with_recorded_python_perf_maps(reader)`	Allow perf maps for PIDs ever recorded as Python runtimes in the spool.
`with_perf_maps(modules, allow)`	Globally enable or disable perf-map lookup.
`with_perf_map_processes(modules, pids)`	Allow perf maps only for the listed PIDs.
`for_each_sample_stack(stack, visit)`	Resolve a `SampleStack` from `sample_stacks()` and stream borrowed resolved frames to `visit`.
`for_each_resolved_frame_slice(pid, frames, visit)`	Resolve a caller-supplied raw-frame slice and stream borrowed resolved frames to `visit`.

for_spool_with_recorded_python_perf_maps is intentionally broader than a “last observed as Python” filter: a PID remains allowed if it was ever marked as a Python runtime in the spool. Use with_perf_map_processes for stricter PID-reuse handling.

Resolution order, top to bottom:

Python or JIT perf map at /tmp/perf-<pid>.map, if allowed and the frame matches.
ELF symbols for file-backed user modules.
Kernel symbol lookup for kernel frames.
Address-only fallback.

§`ResolvedFrame`

Variant	Meaning
`Python(PythonFrame)`	Python frame from a perf-map symbol.
`Native(NativeFrame)`	Native, kernel, JIT, or address-only frame.

ResolvedFrame::func_name() gives you a displayable name for either.

§`PythonFrame`

Field / method	Meaning
`file_name`	Python source filename.
`location`	Line + column when available.
`func_name`	Python function name.
`opcode`	Optional opcode.
`is_entry`	Entry marker?
`basename()`	Filename without leading dirs.

§`NativeFrame` and `NativeSymbol`

NativeFrame:

Field	Meaning
`pc`	Program counter.
`sp`	Stack pointer when available (currently `0` from the public symbolizer).
`symbol`	`Option<NativeSymbol>`. `None` means address-only.
`is_python_runtime`	Belongs to Python runtime machinery.
`kind`	`FrameKind::Native`, `Kernel`, or `Unknown`.
`origin`	`SymbolOrigin`. Where the name came from.
`flags`	`FrameFlags` for UI policy.

NativeSymbol carries the symbol name, optional source file / line, module name, basename offsets, module-relative offset, and Python-runtime helpers like is_eval_frame and should_ignore.

§Kinds, origins, flags

Type	Values
`FrameKind`	`Python`, `Native`, `Kernel`, `Unknown`
`SymbolOrigin`	`Elf`, `PerfMap`, `KernelSymbols`, `AddressOnly`
`FrameFlags`	`PYTHON_RUNTIME`, `HIDDEN_DEFAULT`, `JIT`, `TRUNCATED_STACK`

UIs typically hide HIDDEN_DEFAULT, badge JIT, group by FrameKind, and expose SymbolOrigin in a details view.

§Feature flags

Feature	Effect
`debuginfod`	Enables the default native symbolizer to query debuginfod when `DEBUGINFOD_URLS` is set.

STACKPULSE_DEBUG_DIRS overrides local debug-file search roots. With debuginfod, STACKPULSE_DEBUGINFOD_CACHE_DIR overrides the debuginfod cache directory.

§Process launch and liveness

§`process::SuspendedLaunchedProcess`

Method	What it does
`launch_in_suspended_state(cmd, args, env)`	Fork a child that waits before `execve`.
`pid()`	The child’s PID before it has executed.
`unsuspend_and_run()`	Let it `execve`, returns `process::RunningProcess`.

§`process::RunningProcess`

Method	What it does
`try_wait()`	Non-blocking wait.
`wait()`	Blocking wait until exit.

§`children`

Function	What it does
`discover_all_descendants(root)`	Descendant PIDs via `/proc/<pid>/task//children`, falling back to `/proc//stat`.

§`state`

Function or type	What it does
`ProcessExitWatcher::try_new(pid)`	pidfd-based exit watcher.
`ProcessExitWatcher::poll()`	Non-blocking exit check.
`process_exists(pid)`	Does this PID look alive?
`interrupt_process(pid)`	`SIGINT`.
`kill_process(pid)`	`SIGKILL`.

§Error statistics

SampleErrorStats records per-kind failures. Cloneable, resettable, printable via ErrorStatsFormatter.

Item	What it does
`SampleErrorKind`	Native-unwinding failure kinds (register capture, stack read, framehop errors).
`record(kind)`	Bump a counter.
`record_with_log(kind, ctx)`	Bump and emit a throttled debug log.
`get(kind)`	Read one counter.
`total()`	Sum across kinds.
`has_errors()`	Any non-zero?
`iter_nonzero()`	Iterate the non-zero counters.
`reset()`	Zero everything.
`ErrorStatsFormatter::new(stats, total_samples, successful_samples)`	Build a display formatter.
`write_to(writer)`	Write a grouped report.

§Constants and helpers

Item	Meaning
`MAX_SAMPLE_USER_STACK`	Maximum user stack bytes perf will accept.
`max_sample_rate`	Reads `/proc/sys/kernel/perf_event_max_sample_rate`, `None` if unavailable.
`is_python_module`	Does this basename look like a Python executable or `libpython`?
`path_to_name`	Display name from a path.
`ModuleImageBase`	Translates runtime AVMA addresses to static VMAs.
`PerfFrequencyLimit`	Error payload when requested frequency exceeds the kernel cap.

§Spool format invariants

Append-only and compact:

Modules, frames, stack nodes, threads, samples, and process exec markers are separate record kinds.
Frames are interned. Repeated frames stored once.
Stacks are prefix nodes. Common suffixes shared.
Threads are interned by (process_id, thread_id).
Sample timestamps stored as deltas (ns).
timestamp_us maps perf time to profile time using the stored start timestamp and the first sample.

The on-disk layout is an implementation detail. Read spool files through PerfSpoolReader.

§How it works

This chapter explains the mental model behind stackpulse: how the kernel samples threads, what ends up in the spool, why recording and symbolization are separate phases, and where overhead and dropped frames come from. Read it when something looks off in your profiles and you need to know whether to blame the kernel, the symbolizer, or your own configuration.

§Sampling, not tracing

stackpulse is a statistical sampler. It doesn’t record every function call. The kernel periodically interrupts threads, snapshots enough state to describe where they were, and drops records into perf ring buffers.

If a function shows up in 20% of samples, the right reading is “the program was observed in or below that function about 20% of sampled time”, not a call count. Short functions can be invisible, and brief spikes can be missed if no sample lands on them.

§The pipeline

target threads
  → perf_event_open ring buffers
  → PerfRecorder::consume_available
  → native unwinding + module tracking
  → compact spool file
  → PerfSpoolReader
  → PerfSymbolizer
  → your aggregator / UI / exporter

The split is deliberate. Recording does only what’s needed to preserve the profile. Expensive optional work (symbol lookup, aggregation) happens after the data is safely written.

§A short tour of perf events

perf_event_open is the Linux syscall that exposes the kernel’s Performance Monitoring Unit (PMU) and a handful of software event sources to user space. You ask the kernel “tell me about this event for this task on these CPUs” and get back a file descriptor. The kernel keeps a counter behind that fd and, if you asked it to, also emits a stream of records into a shared ring buffer whenever the event fires.

Two event families matter here:

Hardware events from the CPU’s PMU. The most useful one for profiling is the CPU cycles counter, which ticks whenever the core is running. The PMU is finite (a handful of counters per core) and many distros and virtualization layers restrict access to it.
Software events synthesized by the kernel. The relevant fallback is the CPU clock, a monotonic per-task timer. It doesn’t need PMU hardware, so it works inside containers and VMs where the PMU is hidden.

stackpulse tries hardware CPU cycles first and falls back to the software CPU clock if the kernel refuses or the hardware event isn’t available. Both are CPU-time sources: they tick when a thread is actually on a CPU, so they under-represent time spent blocked on I/O, locks, or sleep. Off-CPU attribution is out of scope here; that needs a different sampling discipline (sched switches, eBPF, or wallclock samplers).

Sampling vs. counting: when you set freq and a sample period, the kernel treats the counter as a target rate and writes one record every time the counter overflows the period. That’s how a frequency-based profiler gets roughly N samples per second per CPU without you knowing the exact cycle count. The kernel adjusts the period over time to keep the rate near the requested frequency.

For each target, stackpulse configures the event to emit:

frequency-based sampling at the requested rate;
monotonic timestamps, so records from different ring buffers can be merged into a single timeline;
task IDs (PID + TID) inside each sample;
the user-mode register set at the moment the sample fired;
a copy of the user-mode stack bytes. The kernel literally memcpys up to stack_size bytes of the user stack into the record;
mmap, comm, fork, and exit side-band records, so we learn about new executable mappings, process names, forks, and exits without re-reading /proc;
lost-event records, so the kernel can tell us when it had to drop samples because we were too slow;
frame-pointer callchains for user fallback frames, with kernel callchains included only when include_kernel is on.

Each event has its own mmap’d ring buffer. The kernel is the producer, we are the consumer, and the two sides coordinate through head/tail pointers in a header page. Because samples can be generated on any CPU, on many CPUs in parallel, records from different buffers don’t arrive in global timestamp order. wait blocks (via epoll on the event fds) until at least one buffer looks readable; consume_available then drains every ready buffer, merges records across them by timestamp, updates the recorder’s view of processes and modules, runs the native unwinder on sample records, and writes the resulting compact records to the spool.

If the consumer can’t keep up, the ring buffer fills, the kernel starts dropping samples, and emits a LOST record so we can count what was lost in PerfSummary.lost_events. That’s the single most important back-pressure signal during recording.

§Attach modes

Two modes cover the practical cases:

StopAttachEnableResume is for an existing process. The recorder briefly SIGSTOPs the target, opens the perf events, registers the executable mappings from /proc/<pid>/maps, enables the events, and resumes the target. The short stop window keeps the initial view of threads and mappings consistent with what perf will see going forward.

AttachWithEnableOnExec is for a forked-but-not-yet-execved child: create the events first, let the kernel turn them on at execve, and nothing is missed during startup.

§Threads vs. child processes

Perf events open against tasks. stackpulse tracks the process leader plus known threads and asks perf to inherit events for new threads. When inheritance isn’t an option, refresh_threads scans /proc/<pid>/task and opens missing ones.

Child processes are not threads. Use inherit_child_processes to follow forks after recording starts. The recorder watches for fork events, clones the relevant module state from the parent, and opens the child. Pre-existing descendants need explicit attachment.

§Native stack capture

For user frames, perf hands us the interrupted thread’s user registers plus a bounded byte copy of the user stack. framehop unwinds from there. Perf also supplies user frame-pointer callchains, which are used only as a fallback when DWARF unwinding cannot produce the deeper frames.

Stack-copy size is a trade-off. Too small and unwinding stops short, and PerfSummary.error_stats shows truncation. Too large and every sample copies more memory than necessary, which raises overhead at the same sampling rate. Starting around 60 * 1024 and adjusting based on the summary counters works for most workloads.

Return-address frames are normalized: each return address is rewound to the instruction before the return target so symbol lookup lands on the call site, not the next instruction after.

§Kernel frames

The recorder asks perf for frame-pointer callchains and uses them for kernel frames when include_kernel is enabled. User frames still prefer the native DWARF unwinder; perf’s user frame-pointer frames are kept as a fallback when DWARF unwinding stops early or produces no frames. Any user frame-pointer frames left unused are counted in ignored_user_callchain_frames.

Kernel sampling is usually permission-gated. If perf_event_open fails only because kernel sampling was denied, attach retries without kernel frames and reports kernel_enabled = false.

Kernel names come from /proc/kallsyms when it’s readable and usable; otherwise kernel frames render with an address-based name.

§Module tracking

A raw IP isn’t enough to symbolize a frame. The resolver needs to know which mapping owned that address and how the mapping ties to its backing file.

Mappings come from two places:

the snapshot of /proc/<pid>/maps taken at attach;
perf mmap records emitted while the process runs.

Each mapping becomes a ModuleRecord with its runtime address range, file offset, inode, path, owning PID, and kernel flag. The recorder resolves each frame’s absolute address to a module ID plus a module-relative IP when possible, so symbolization doesn’t need the target process to still exist.

§Symbolization

PerfSymbolizer resolves frames after the fact, from several sources:

Source	Used for	Result
Python perf maps (`/tmp/perf-<pid>.map`)	Python frames and JIT-like symbols emitted by runtimes.	`PythonFrame`, or `NativeFrame` with `SymbolOrigin::PerfMap`.
ELF + debug data	Native user-space modules. Routed through a pluggable `NativeSymbolizer`; default is wholesym.	`NativeFrame` with `SymbolOrigin::Elf`.
`/proc/kallsyms`	Kernel frames.	`NativeFrame` with `FrameKind::Kernel`.
Address fallback	No symbols or mapping unknown.	`NativeFrame` with `SymbolOrigin::AddressOnly`.

Python frames exist only when the runtime emits perf-map entries. For modern CPython, -X perf or PYTHONPERFSUPPORT=1. The recorder writes process exec markers so readers can restrict perf-map lookup to PIDs that actually looked like Python runtimes during recording.

The spool file does not embed perf-map content. Symbolization reads the on-disk /tmp/perf-<pid>.map. If you’ll analyze later, on another host, or after the process exits, preserve those map files next to the spool.

Native frames inside the Python runtime get FrameFlags::PYTHON_RUNTIME and FrameFlags::HIDDEN_DEFAULT when the symbolizer can identify them. UIs can hide interpreter machinery by default while still letting users dig in.

Native symbolization is delegated to a NativeSymbolizer implementor, one per non-overlapping module group. The default is the bundled wholesym backend, configured from STACKPULSE_DEBUG_DIRS, DEBUGINFOD_URLS, and related environment variables. Embedders with their own debuginfod, debug-dir, or source-info pipeline can swap that backend out through PerfSymbolizer::with_native_factory, and PerfSymbolizer keeps owning kernel-frame and perf-map resolution. Each SymModule handed to the plug-in already carries a resolved ModuleImageBase, so the plug-in only needs to parse ELF for symbol lookup, not for layout.

§Why spool files are small

Profiles repeat themselves. Hot loops produce the same frames and stacks many times. The format exploits that:

module records are written once when a mapping is discovered;
thread IDs are interned;
frame records are interned;
stacks are stored as prefix nodes so common suffixes are shared;
samples point to a thread ID and a stack ID;
timestamps are stored as deltas.

Writes stay small and repeated stacks are cheap. PerfSpoolReader expands stack IDs back into frame records when an analysis needs them.

§Accuracy and bias

Sampling has predictable limits:

It records where threads were when samples fired, not every call.
CPU-time sources under-represent off-CPU work (I/O, locks, sleep).
Very high frequencies can lose events if buffers aren’t drained fast enough.
Unwinding can fail when stack bytes are short, metadata is missing, or the thread is in a hard-to-unwind state.
Symbol quality depends on binaries, debug info, perf maps, kernel symbol visibility, and whether the mappings were observed.
PID reuse makes stale /tmp/perf-<pid>.map files dangerous unless lookup is restricted to PIDs whose latest exec marker says they’re Python.

The PerfSummary counters exist to make those limits visible. A profile is only as trustworthy as those numbers say it is: check sample count, lost events, empty stacks, truncation markers, and error stats before drawing conclusions from a recording.

§Overhead

Recording costs:

kernel interrupt + sample collection at the requested frequency;
copied user stack bytes per sample;
ring buffer traffic;
native unwinding in consume_available;
spool writes;
extra events for many threads, CPUs, or inherited children.

Symbolization is intentionally off the hot path. ELF data, debug info, kernel symbols, and perf maps are read lazily after recording.

To trim overhead: lower frequency, lower stack_size, skip kernel frames unless you need them, limit child-process inheritance, and drain often enough from a dedicated worker that you don’t lose events.

§Permissions

Linux perf access is gated by the kernel and by distro policy. The usual gates:

ownership of the target process;
/proc/sys/kernel/perf_event_paranoid;
/proc/sys/kernel/perf_event_max_sample_rate;
capabilities such as CAP_PERFMON (or full admin on older kernels);
/proc/<pid> visibility inside containers and PID namespaces;
read access to /proc/kallsyms for kernel symbol names.

Plan for graceful degradation. User-space capture without kernel frames is usually still useful, and address-only frames remain useful as long as you can symbolize them later against the same binaries.

Re-exports§

pub use profile::FrameFlags;
pub use profile::FrameKind;
pub use profile::LocationInfo;
pub use profile::NativeFrame;
pub use profile::NativeSymbol;
pub use profile::PythonFrame;
pub use profile::ResolvedFrame;
pub use profile::SourceLocation;
pub use profile::SymbolOrigin;

Modules§

children: Helpers for discovering and following child processes spawned by a target.
process: Spawn and attach helpers for the target process.
profile: Post-symbolization frame model returned by PerfSymbolizer.
state: Process-state snapshots used by the recorder to translate kernel events.

Structs§

ErrorStatsFormatter: Format error statistics for display.
FrameContext: Raw frame plus its recorded module context, when Stackpulse had one.
FrameModuleRef: Recorded module context for a raw frame.
FrameRecord: A raw frame stored in a profile file.
ModuleImageBase: The image-wide base addresses for one loaded object.
ModulePath: File path or display name for a recorded module.
ModuleRecord: A code area recorded in a profile file.
OwnedSampleRecord: A sample record loaded from a profile file.
PerfFrequencyLimit: Error returned when the requested sample rate exceeds the kernel’s perf_event_max_sample_rate.
PerfRecorder: Records stack samples for one or more Linux processes.
PerfRecorderOptions: Options used when attaching a PerfRecorder to a process.
PerfSpoolReader: Reader for profile files written by crate::PerfRecorder.
PerfSummary: Counters collected while recording.
PerfSymbolizer: Resolves raw profile frames into displayable frames.
ProcessExecRecord: Marker for a process that executed during recording.
SampleErrorStats: Atomic counters for sample error statistics.
SampleStack: Borrowed sample and its no-copy raw stack iterator.
SampleStacks: No-copy iterator over all samples and their raw stacks.
StackFrameContexts: Borrowed raw frames with recorded module context for one interned stack.
StackFrameRefs: Borrowed raw frames for one interned stack.
SymModule: Module information for symbolization.

Enums§

AttachMode: How recording should attach to a process.
Error: Error type returned by fallible stackpulse APIs.
FrameMode: Whether a frame came from user code or kernel code.
SampleErrorKind: Categories of sample failures for statistics tracking.

Constants§

MAX_SAMPLE_USER_STACK: Hard kernel cap on the user-stack snapshot size, in bytes, that perf_event_open will copy per sample. Acts as a ceiling for PerfRecorderOptions::stack_size; anything larger is rejected.

Traits§

NativeSymbolizer: Plug-in interface for native (ELF/Mach-O) module symbolization.

Functions§

default_native_symbolizer_factory: Default factory: returns stackpulse’s bundled wholesym-backed SymbolizerWrapper, configured from STACKPULSE_* env vars.
is_python_module: Heuristic check for whether a module basename belongs to a Python runtime.
max_sample_rate: Read the kernel’s current maximum perf sample rate, in samples per second.
path_to_name: Display-friendly basename for a module path.

Type Aliases§

NativeSymbolizerFactory: Factory that produces a NativeSymbolizer for a given process id. PerfSymbolizer calls this once per non-overlapping module group.
Result: Convenience alias for Result<T, Error> used throughout the crate.
SymbolsRc: Cached symbols - wrapped in Rc for cheap cloning

Crate stackpulse Copy item path

§Quick example

§Core types

§Raw replay

§Plugging in an external native symbolizer

§Vocabulary

§Runtime requirements

§Tutorials

§Attach to an existing process

§Capture process startup

§Aggregate into a stack histogram

§Recipes

§Pick recording options

§Drain without dropping samples

§Profile more than one process

§Follow children created after recording starts

§Catch threads created later

§Resolve symbols

§Use your own symbolizer

§Python frames

§Kernel frames

§Diagnose bad profiles

§Permission failures

§Reference

§Module map

§Recording

§Reading spool files

§Symbolization

§NativeFrame and NativeSymbol

§Kinds, origins, flags

§Feature flags

§Process launch and liveness

§Error statistics

§Constants and helpers

§Spool format invariants

§How it works

§Sampling, not tracing

§The pipeline

§A short tour of perf events

§Attach modes

§Threads vs. child processes

§Native stack capture

§Kernel frames

§Module tracking

§Symbolization

§Why spool files are small

§Accuracy and bias

§Overhead

§Permissions

Re-exports§

Modules§

Structs§

Enums§

Constants§

Traits§

Functions§

Type Aliases§

Crate stackpulse

§`NativeFrame` and `NativeSymbol`