# fast-observe — design
> Design rationale and invariants. The implemented user surface is documented in rustdoc and OBSERVE.md; where this doc and the code disagree, the code wins.
Goal: one crate that deploys errors-with-causal-trees, profiling, logs, and
traces sharing identifiers — fastrace + logforth + a multi-profiler facade
fused with an exn-grade typed error tree, producing causal, prescriptive,
LLM-agent-readable output.
Non-goals: reimplementing profiling backends, replacing tracing, being a
metrics system.
---
## 0. The thesis
Three facts drive the design:
1. **`profiling` crate's compile-time model is right for heavy profilers**
(puffin/optick/tracy/superluminal/tracing): zero code when off, native
instrumentation when on. Runtime selection is wrong for these — tracy
either is linked or is not.
2. **Runtime selection is right for the cheap built-in backends**
(Off/Instant/Fastrace/Web): they are always compiled in, cost ~2ns when
off, and switching at runtime via env is a genuine dev workflow.
3. **The error path is the product.** Profiling and logging are sinks; the
`Fault` causal tree is the data model. Everything else (hooks, scopes,
trace ids, diagnostics, registry) exists to make the final error report
more causal and more actionable.
So: keep runtime selection for Tier-1 backends, feature-forward Tier-2
backends to upstream `profiling`, and invest the design budget in the error
data model + the report.
---
## 1. Layering (hard dependency rule)
```
layer 0: core Fault / Frame / Context / define_errors! / ERROR_REGISTRY
error_counts / hooks — depends on: parking_lot, log (facade only)
layer 1: observe profiling facade, config, diagnostics (ariadne), report
layer 2: deploy init()/builder wiring logforth + fastrace + Tier-2 profilers
```
Rule: **lower layers never name logforth/fastrace types**. Layer 0 must work
pre-`init()`, post-`init()`, in tests, in wasm, with hooks cleared. The error
path never breaks because a sink is missing. This is already mostly true
(hook.rs is the only place logforth/fastrace are named); the Deployment
builder does not regress it.
Single crate, feature-gated. Not a workspace: the layers are small, and
feature flags give identical pay-for-what-you-use granularity without
version-skew pain.
---
## 2. Pillar: profiling facade (two tiers)
### Tier 1 — runtime-selected (existing, keep)
`ObserveConfig.profiling_backend: Off | Instant | Fastrace | Web`,
`OBSERVE_PROFILE` env. `scope!` = one relaxed atomic load + branch.
Fixes folded in:
- `scope!` currently checks `!= Off` in the macro and re-matches in
`ScopeGuard::new_static` — fold to one load.
- `ScopeGuard::new` (dynamic names) `Box::leak`s — replace with a global
intern table (`Mutex<HashSet<&'static str>>`), bounded by unique names.
- `CURRENT_SCOPE` stores `(Cow, Instant)`; the `Instant` is never read.
Use it: `Fault::new` attaches `scope elapsed` to the frame context.
- **Clock: adopt fastant on native, keep web-time on wasm.**
MIGRATING.md §7 dropped the old rdtsc clock for `web_time::Instant`
(~tens of ns per read on native) purely for wasm portability. fastant
(same `fast` org as fastrace/logforth; fastrace times its own spans with
it) gives TSC reads (~2–5ns) on Linux x86_64 and falls back to
`std::time` elsewhere — a superset of today's native behavior. Split by
target, keep `clock.rs` `pub(crate)` so the swap is invisible:
```toml
[target.'cfg(not(target_family = "wasm"))'.dependencies]
fastant = "0.1"
# web-time stays, wasm-only
```
- native: `fastant::Instant` (TSC; auto-fallback if TSC unstable)
- wasm: `web_time::Instant` (fastant would fall back to std, which
panics on wasm32-unknown-unknown — the original reason for web-time)
- feature-forward `clock-coarse` → `fastant/fallback-coarse` for
speed-over-accuracy fallback platforms
- bonus beyond speed: instant-backend spans and fastrace spans now share
one clock source + calibration, so durations are directly comparable
across the two Tier-1 backends (the isomorphism rule applied to time).
### Tier 2 — runtime-selectable backend SET (REVISED; no upstream `profiling` dep)
Original plan forwarded to the upstream `profiling` crate. REJECTED:
upstream's model is compile-time-only (feature on = always instrumenting),
and the design decision here is that **compiled-in ≠ active** — backends
are compiled in via features and SELECTED at runtime via flags, alongside
Tier-1. (User directive: "no profiles; just selecting what you want and
having flags/toggles. Default is fastrace; you turn instant on to look at
performance.")
Instead we own ~15-line glue modules per backend (upstream `profiling` is
the reference implementation — same APIs, same semantics), each behind
`profile-with-*` features named verbatim after upstream:
```toml
profile-with-puffin = ["dep:puffin"]
profile-with-tracy = ["dep:tracy-client"]
profile-with-optick = ["dep:optick"]
profile-with-superluminal = ["dep:superluminal-perf"] # windows-only dep
profile-with-tracing = ["dep:tracing"] # our scope! → tracing::span
```
Runtime selection replaces `ProfilingBackend` with a bitmask:
```rust
```
- `scope!` = ONE atomic load of the mask; each enabled+compiled backend
enters its guard (ZST stubs when not compiled — the existing
`profiling_backend!` wrap-module pattern extended per backend).
Mask 0 → all-dummy guard, ~2ns.
- `OBSERVE_PROFILE` becomes a comma list (`fastrace,tracy`); single names
(`off|instant|fastrace|web`) keep working.
- **Self-teaching config**: setting a bit whose feature is not compiled
logs a one-time warning naming the exact cargo feature to enable
(`profile-with-tracy`). LLM agents reading the log learn the flag.
- Backend enable hooks: puffin needs `set_scopes_on(true)` —
`set_backends` calls per-backend `on_enable()` when a bit flips on.
- Dependency weight is the whole point of features (see §9d): tracy/optick
compile C/C++, otel pulls the OTel SDK tree — none of it exists in the
build unless its feature is named.
Attribute-macro re-export: superseded by §9c-ext — the macros are owned by
fast-observe-macros, not re-exported from `profiling-procmacros`.
### Async gap
`scope!` guards are thread-bound (`!Send`); instrumenting across `.await`
needs a dedicated surface. With feature `int-futures` (fastrace-futures):
```rust
future.in_observed_span("load") // wraps fastrace's in_span + scope-name TLS
```
and document: sync scopes inside async fns are fine between awaits;
cross-await spans must use `in_observed_span` or a root span.
---
## 3. Pillar: log pipeline (full logforth surface + typestate builder)
### Feature map (all optional; `bridge-log`, `fastrace` stay default)
| `layout-json` (rename of `json`, alias kept) | JSON stdout |
| `layout-logfmt` / `layout-gcl` / `layout-text` | logfmt / Google Cloud / plain text layouts |
| `log-stderr` | Stderr appender (error split) |
| `log-file` (rename of `file`, alias) | rolling file via OBSERVE_LOG_DIR |
| `log-syslog` / `log-journald` | system log sinks |
| `log-async` | Async combiner appender (offload formatting/IO) |
| `log-testing` | capture appender for harnesses |
| `filter-rustlog` | `OBSERVE_LOG`/`RUST_LOG` directive filtering |
| `diag-task-local` | task-local MDC for async |
| (existing) `otel` | OTel appender + reporter |
### Builder (typestate where it matters)
```rust
let guard = fast_observe::builder()
.logs(|l| l
.env_filter() // OBSERVE_LOG, fallback RUST_LOG, default "info"
.stdout() // text; colors iff tty && !NO_COLOR
.stderr_from(Level::Error) // optional split
.file_from_env()) // OBSERVE_LOG_DIR when log-file
.traces(|t| t.console()) // .reporter(r) / .otel(...) / .off()
.errors(|e| e.throttle(100).backtrace(true))
.init()?; // Err(InitError) on double-init
```
Type system doing real work, not ceremony:
- `init()` is the only terminal and returns
`Result<InitGuard, InitError>`; `InitGuard: Drop` calls `fastrace::flush()`
— kills the "forgot to flush, lost the last trace" class of bug.
- Reporter handling: `set_reporter` is global and unstoppable; the builder
tracks `reporter: Option<...>` and refuses to stomp unless `.force()` —
the Option in the type makes the conflict explicit.
- `init()` zero-arg keeps working: `builder().init()` with all defaults,
so README's one-liner survives.
- `fastrace::flush` on guard drop is best-effort; documented.
### Env surface (single `EnvConfig`, parsed once in the LazyLock)
`OBSERVE_PROFILE` (have), `OBSERVE_LOG`, `OBSERVE_LOG_DIR` (have),
`OBSERVE_ERROR_THROTTLE`, `OBSERVE_BACKTRACE`, `OBSERVE_COLOR=always|never|auto`.
Unknown values warn-and-default (existing pattern). Every knob documented in
one table in README.
### Hooks
`add_error_hook` stays; `clear_error_hooks()`, `hooks_len()` (tests), and
`set_default_hook_enabled(bool)` round out hook management. Hook list clone-per-error is fine (cold
path); if a profile ever says otherwise, swap `Mutex<Vec<Hook>>` for
`arc_swap::ArcSwap<Vec<Hook>>` — internal change, no API impact.
---
## 4. Pillar: fastrace surface
- Reporter pluggability via builder (above); `init_otel` folds into
`.traces(|t| t.otel(reporter))`.
- Re-export features (mirrors upstream integration crates):
`int-futures`, `int-axum`, `int-poem`, `int-tonic`, `int-tower`;
existing `http` (reqwest) and `bridge-tracing` unchanged.
- Root-span ergonomics: `root_span!("request")` macro →
`Span::root(func_path!()-aware name, SpanContext::random())` +
`set_local_parent` guard. Today every app hand-rolls this; it is the
difference between "trace exists" and "trace correlates with logs/errors".
- `fast_observe::flush()` re-export (calls fastrace::flush when enabled,
no-op otherwise).
---
## 5. Pillar: exn-core — the typed error data model
The attraction of exn is that context is *typed structure*, not strings.
Push that further than exn does:
### 5.1 Attachments — SUPERSEDED by §5.7 (typed + placement, one design)
Original two-channel sketch kept for reference:
```rust
pub struct Attachment { pub key: &'static str, pub value: Cow<'static, str> } // rendered
struct TypedSlot(Box<dyn Any + Send + Sync>); // programmatic
impl Fault<E> {
fn attach(self, key: &'static str, value: impl Display) -> Self; // render channel
fn attach_typed<T: Send + Sync + 'static>(self, v: T) -> Self; // type channel
fn get<T: 'static>(&self) -> Option<&T>;
}
```
Render channel prints `[key=value ...]` in Display/Debug; type channel is
the error-stack-style grab bag for programmatic recovery (retry hints,
partial state). Both live on the root frame; `wrap` keeps the child's
attachments reachable via the tree.
### 5.2 Traversal + typing across the tree
- `Fault::iter()` — preorder `&Frame` iterator; `root_cause()`;
`Frame::find_type(name)` for doctor tooling.
- `walk_sources` currently FLATTENS `A→B→C` into siblings of root and
stringifies each into `InternalError`, and `wrap_msg` forgets to walk at
all. Fix: nest properly (`B` child of `A`, `C` child of `B`), keep
`type_name` per frame, and route ALL construction through one internal
`Frame::capture(...)` so `new`/`wrap`/`wrap_msg`/`observed`/`from_boxed`
cannot drift (they already have: `wrap_msg` drops the source chain).
### 5.3 Codes and categories as types
- `define_errors!` types implement a new `Coded { fn code(&self) -> &'static str }`
trait; `write_fault` renders `[E100]` in the tree when the frame's error
is coded (via a `&dyn Coded` downcast slot stored at capture time).
- `ErrorCategory::policy()` → `Policy::{Retry{after}, Poison, Abort,
ContentFix}` — category stops being a label and becomes behavior; the
report renderer prints the policy as the prescriptive line.
- `error_counts_by_category()` — registry lookup per type → grouped counts.
### 5.4 Boundaries
Feature `anyhow-boundary`: `from_anyhow(anyhow::Error) -> Fault<AnyhowError>`,
`into_anyhow(Fault<E>) -> anyhow::Error` (exn-anyhow pattern; explicit
`map_err` at the API boundary, never implicit).
### 5.5 Capture
Feature `backtrace` + `OBSERVE_BACKTRACE=1`:
`Backtrace::force_capture()` stored on the root frame, rendered in Debug
tree. Off by default (cost), on in dev via env. Location-per-frame stays
(always on, free).
### 5.6 Two hook families (rootcause's best idea, adapted)
Rootcause splits hooks into *creation* hooks (mutate the report at
capture time: attach backtrace, span fields, request ids) and *formatting*
hooks (control rendering). Our single `add_error_hook` is neither — it is a
read-only *sink* fan-out (log/metrics notification). All three jobs are
distinct; model them as three registries:
1. **Capture hooks** (new): `add_capture_hook(fn(&mut FrameCapture))` —
run DURING `Frame::capture`, can attach data. Backtrace capture,
trace-id, scope-elapsed, hostname/pid all become capture hooks instead
of hardcoded fields — features register them, apps add their own
(request id!). `OBSERVE_BACKTRACE` toggles the built-in one.
2. **Sink hooks** (existing `add_error_hook`): post-construction fan-out,
throttled, panic-contained. Unchanged semantics.
3. **Formatter** (new, §7): the report renderer becomes a replaceable
global formatter (`set_report_formatter`) — text default, `json` under
serde, app-defined layouts. Rootcause splits formatting into four hook
types (report/context/attachment + placement); we take ONE
report-level formatter + per-attachment placement (below), not the
full matrix.
### 5.7 Attachments upgraded: typed + inspectable + placement
Rootcause proves attachments should not be "glorified strings": their
`attachment.downcast_inner::<T>()` makes reports programmatically
inspectable (their example: extracting `RetryMetadata` from a retry tree).
Our planned two-channel attachment merges into ONE typed design:
```rust
pub struct Attachment {
value: Arc<dyn Any + Send + Sync>, // typed channel: get::<T>()
display: AttachmentDisplay, // render channel: cached Display string
placement: Placement, // Inline | Appendix | Opaque | Hidden
}
impl Frame {
pub fn attach<A: Display + Send + Sync + 'static>(&mut self, a: A); // typed+rendered
pub fn attachments(&self) -> &[Attachment];
pub fn find_attachment<T: 'static>(&self) -> Option<&T>;
}
```
`attach("key", value)` string form remains as sugar over this. Placement
controls the §7 report layout (Opaque = counted-not-shown, for secrets/
large payloads — rootcause's redaction answer). Keyed lookup stays via the
string form; typed lookup via `find_attachment`.
### 5.8 Multi-failure: `FaultCollection`
Rootcause's headline feature beyond anyhow: retry/batch failures collected
into one tree (`ReportCollection::new(); errors.push(e);
Err(errors.context("..."))`). Our Frame tree already supports n-ary
children; add the ergonomic surface:
```rust
let mut errs = FaultCollection::new();
for attempt in 1..=3 {
match fetch().attach_with(|| format!("attempt #{attempt}")) {
Ok(v) => return Ok(v),
Err(e) => errs.push(e),
}
}
Err(errs.into_fault(JournalError::Flaky)) // one Fault, three children
```
Retry/recovery loops and batch-compaction paths are the consumers.
### 5.9 Compat matrix (rootcause's full set, same pattern per crate)
Features `compat-anyhow`, `compat-eyre`, `compat-error-stack`, plus
always-on `Box<dyn Error>` conversion. Bidirectional, explicit, tree
preserved both directions. `into_fault()` / `from_fault()` extension
trait methods (their `IntoRootcause` pattern) so call sites read
`legacy_fn().into_fault()?`.
---
## 6. Pillar: diagnostics (ariadne, done properly)
1. **Multi-label**: `Diagnostic.labels: Vec<LabelSpan>` with
`{ span, message, primary, color? }`; `with_source` stays as the
one-label convenience; new `with_label(...)`.
2. **In-memory sources**: global `SourceStore`
(`RwLock<HashMap<Utf8PathBuf, Arc<str>>>`); the ariadne cache checks the
store, then disk. Diagnostics for generated/embedded/packed assets stop
rendering `<unknown>`.
3. **Registry linkage**: `build_report` looks up `diag.code` in
`ERROR_REGISTRY`; on hit, appends
`note: [E100] disk read failed (category: Transient — safe to retry)`.
The compiler-error path and the runtime-error path finally share the
same code registry.
4. `Diagnostic::warning()/info()` constructors; `impl Error for Diagnostic`;
`eprint_diagnostic` honors `NO_COLOR` + tty detection.
5. Rejected: miette interop. ariadne is already the renderer and miette's
protocol would fork the data model. Revisit only if ecosystem pressure
demands it.
---
## 7. Pillar: the Report — LLM-agent-first output
New module `report`: `render_report(&Fault<E>) -> String`
(+ `render_report_json` under `serde`).
Design principles (this is the "knowing LLM agents read it" part):
- **Stable sections, one fact per line, `key: value`.** No prose
paragraphs. Diff-stable = snapshot-testable = reliably parseable by an
agent with no special tooling.
- **No ANSI, no wall-clock timestamps** (elapsed/durations only), absolute
paths — output identical across machines.
- **Deterministic order**: error → location → scope → attachments → cause
chain → trace → action.
- **Prescriptive final line** derived from category policy — the agent's
next action is in the output, not in tribal knowledge.
Format sketch:
```
error: [E100] disk read failed: /etc/app/spec.toml
category: Transient (policy: retry is safe)
location: src/config.rs:42:10
scope: load_config (elapsed 12.3ms)
attachment: attempt=3
attachment: path=/etc/app/spec.toml
cause 0: disk read failed: /etc/app/spec.toml
cause 1: No such file or directory (os error 2)
trace_id: 4f3c…9a2b # grep this: logs + spans carry the same id
action: retry the operation; if persistent, check OBSERVE_LOG output for trace_id 4f3c…9a2b
```
The `trace_id` line is the keystone: the error hook reads the current
fastrace `SpanContext` at capture time; `FastraceDiagnostic` already stamps
logs with the same id; one grep across `app.log` reconstructs the entire
causal moment — error tree, log lines, span timings. That is "making them
work well together" made concrete.
Default hook gains an option to emit this block (structured, one event)
instead of today's single `log::error!` line.
---
## 8. Ergonomics: the whole API in one import
```rust
pub mod prelude {
pub use crate::{Result, Fault, Context, ResultExt, OptionExt, ErrorExt};
pub use crate::{bail, ensure, scope, profiling, finish_frame, root_span};
pub use crate::{add_error_hook, error_counts, lookup_error};
pub use profiling_procmacros::{function, all_functions, skip};
}
```
Dev experience target — three lines to full observability:
```rust
use fast_observe::prelude::*;