pub struct ProfilingTagLayer;inert: per-span pyroscope tagging leaked memory and emptied profiles; correlation returns with the OTLP profiles exporter
Expand description
Inert. Was: a tracing layer that tagged the running pyroscope agent with
trace_id/span_id on every span enter and exit.
§Why this does nothing
Per-span tagging is not implementable against the pprof backend, and
enabling it was strictly worse than having no correlation at all. In
pyroscope-rs the backend’s own comment calls dump_report a “workaround
for pprof-rs to interrupt the profiler”, and both Backend::add_tag and
Backend::remove_tag call it unconditionally. One dump_report symbolises
every entry currently in the collector — backtrace::resolve per frame,
building a fresh Vec<Vec<Symbol>> with a String and a PathBuf per
symbol — and then clears the collector.
So each span enter cost two full profile rebuilds, and each exit two more. Measured on a static-musl build at the shipped 100 Hz sample rate, driving spans from two threads:
t=10s dumps=868726 clears=868725 sessions=1 collector_entries=0~87,000 profile rebuilds per second against one 10-second upload. Two consequences, and the second is why this is not merely a tuning problem:
- The allocation churn grew RSS without bound. Over 90 minutes on static musl the leak was 7.4 MiB/h with this layer and 1.0 MiB/h without, which matches the 6.5 MiB/h observed on brefwiz-spiffe in production, where it OOM-killed both replicas roughly every 5.5 hours against a 512Mi cgroup.
collector_entries=0at every upload: the collector was cleared far faster than the 100 Hz sampler could fill it, so the profiles this layer existed to enrich were arriving essentially empty. The correlation feature destroyed the very data it annotated.
Sampling itself is not implicated — with tagging off, 63,254 samples over 420 seconds moved RSS by 0.03 MiB/h.
§What replaces it
Nothing, on this backend: there is no bounded form. The cost is per tag call, so sampling spans or tagging only roots still buys full profile rebuilds at a fraction of the span rate, and each rebuild still truncates the sample window. Trace/profile correlation returns with the native OTLP profiles exporter this bridge is already sunset-bound against.
The type is kept, registered, and doing nothing so the subscriber stack and the public surface are unchanged; it is removed in the next major.
Trait Implementations§
Source§impl<S> Layer<S> for ProfilingTagLayerwhere
S: Subscriber + for<'a> LookupSpan<'a>,
Available on crate feature profiling-bridge-pyroscope-rs only.
impl<S> Layer<S> for ProfilingTagLayerwhere
S: Subscriber + for<'a> LookupSpan<'a>,
profiling-bridge-pyroscope-rs only.Source§fn on_register_dispatch(&self, subscriber: &Dispatch)
fn on_register_dispatch(&self, subscriber: &Dispatch)
Subscriber. Read moreSource§fn register_callsite(&self, metadata: &'static Metadata<'static>) -> Interest
fn register_callsite(&self, metadata: &'static Metadata<'static>) -> Interest
Subscriber::register_callsite. Read moreSource§fn enabled(&self, metadata: &Metadata<'_>, ctx: Context<'_, S>) -> bool
fn enabled(&self, metadata: &Metadata<'_>, ctx: Context<'_, S>) -> bool
true if this layer is interested in a span or event with the
given metadata in the current Context, similarly to
Subscriber::enabled. Read moreSource§fn on_new_span(&self, attrs: &Attributes<'_>, id: &Id, ctx: Context<'_, S>)
fn on_new_span(&self, attrs: &Attributes<'_>, id: &Id, ctx: Context<'_, S>)
Attributes and Id.Source§fn on_record(&self, _span: &Id, _values: &Record<'_>, _ctx: Context<'_, S>)
fn on_record(&self, _span: &Id, _values: &Record<'_>, _ctx: Context<'_, S>)
Id recorded the given
values.Source§fn on_follows_from(&self, _span: &Id, _follows: &Id, _ctx: Context<'_, S>)
fn on_follows_from(&self, _span: &Id, _follows: &Id, _ctx: Context<'_, S>)
span recorded that it
follows from the span with the ID follows.Source§fn on_event(&self, _event: &Event<'_>, _ctx: Context<'_, S>)
fn on_event(&self, _event: &Event<'_>, _ctx: Context<'_, S>)
Source§fn on_enter(&self, _id: &Id, _ctx: Context<'_, S>)
fn on_enter(&self, _id: &Id, _ctx: Context<'_, S>)
Source§fn on_exit(&self, _id: &Id, _ctx: Context<'_, S>)
fn on_exit(&self, _id: &Id, _ctx: Context<'_, S>)
Source§fn on_close(&self, _id: Id, _ctx: Context<'_, S>)
fn on_close(&self, _id: Id, _ctx: Context<'_, S>)
Source§fn on_id_change(&self, _old: &Id, _new: &Id, _ctx: Context<'_, S>)
fn on_id_change(&self, _old: &Id, _new: &Id, _ctx: Context<'_, S>)
Source§fn and_then<L>(self, layer: L) -> Layered<L, Self, S>
fn and_then<L>(self, layer: L) -> Layered<L, Self, S>
Layer, returning a Layered
struct implementing Layer. Read moreSource§fn with_subscriber(self, inner: S) -> Layered<Self, S>where
Self: Sized,
fn with_subscriber(self, inner: S) -> Layered<Self, S>where
Self: Sized,
Layer with the given Subscriber, returning a
Layered struct that implements Subscriber. Read moreSource§fn with_filter<F>(self, filter: F) -> Filtered<Self, F, S>
fn with_filter<F>(self, filter: F) -> Filtered<Self, F, S>
Auto Trait Implementations§
impl Freeze for ProfilingTagLayer
impl RefUnwindSafe for ProfilingTagLayer
impl Send for ProfilingTagLayer
impl Sync for ProfilingTagLayer
impl Unpin for ProfilingTagLayer
impl UnsafeUnpin for ProfilingTagLayer
impl UnwindSafe for ProfilingTagLayer
Blanket Implementations§
Source§impl<T> BorrowMut<T> for Twhere
T: ?Sized,
impl<T> BorrowMut<T> for Twhere
T: ?Sized,
Source§fn borrow_mut(&mut self) -> &mut T
fn borrow_mut(&mut self) -> &mut T
impl<ST, DT> CastableFrom<ST, Initialized, Initialized> for DT
impl<ST, DT> CastableFrom<ST, Uninit, Uninit> for DT
Source§impl<T> FutureExt for T
impl<T> FutureExt for T
Source§fn with_context(self, otel_cx: Context) -> WithContext<Self> ⓘ
fn with_context(self, otel_cx: Context) -> WithContext<Self> ⓘ
Source§fn with_current_context(self) -> WithContext<Self> ⓘ
fn with_current_context(self) -> WithContext<Self> ⓘ
Source§impl<T> Instrument for T
impl<T> Instrument for T
Source§fn instrument(self, span: Span) -> Instrumented<Self> ⓘ
fn instrument(self, span: Span) -> Instrumented<Self> ⓘ
Source§fn in_current_span(self) -> Instrumented<Self> ⓘ
fn in_current_span(self) -> Instrumented<Self> ⓘ
Source§impl<T> IntoRequest<T> for T
impl<T> IntoRequest<T> for T
Source§fn into_request(self) -> Request<T>
fn into_request(self) -> Request<T>
T in a tonic::Request