# Per-Processor Statistics
rsconstruct shows several "per-processor" or "per-analyzer" statistics tables
(`cache stats`, `analyzers stats`, `graph stats`, `build` summaries). These all
look similar on the surface, but the **data source differs**, and that changes
what we can cheaply show.
This document explains:
1. The three data sources that feed per-X statistics.
2. The per-processor grouping problem in `cache stats`.
3. Options for fixing it, with tradeoffs.
4. Secondary cleanup — graph-level helpers.
## The three data sources
| "How many products does pylint have in this build config?" | graph (in-memory) | free |
| "How many products were built / skipped / restored this run?" | executor stats (in-memory) | free |
| "How many files did each analyzer find?" | `.rsconstruct/deps.redb` (on disk, keyed by analyzer) | fast — single DB scan, key is already the analyzer name |
| "How big is my on-disk cache, per processor?" | `.rsconstruct/cache/descriptors/` (on disk) | **see below — this is the problem** |
### Graph (in-memory, rebuilt each run)
Every `Product` carries its `processor: String` field. Grouping is a simple
iteration over `Vec<Product>`, constructing a `HashMap<String, T>` on the spot.
Every caller that wants per-processor stats does this inline — see
`builder/graph.rs:111`, `builder/build.rs:323,436,467`,
`executor/execution.rs:180,479,524,540`.
### Analyzer dependency cache (`deps.redb`)
The redb schema stores each entry keyed by (source path → dependencies) and
tagged with the analyzer that produced it. `DepsCache::stats_by_analyzer()`
scans the DB once and returns `HashMap<analyzer, (file_count, dep_count)>`.
Grouping is effectively free because the analyzer name is a first-class field.
### Object-store descriptors (`.rsconstruct/cache/descriptors/`)
Each descriptor file is a small JSON blob describing one cached product — its
outputs, their checksums, etc. The **filename is a hash** of the product's
cache key; the file's location tells us nothing about which processor created
it.
Today's code in `object_store/management.rs:169`:
```rust
pub fn stats_by_processor(&self) -> BTreeMap<String, ProcessorCacheStats> {
// walk every file in descriptors_dir
// read the file
// parse the JSON
// ...
// "We can't extract processor name from a hashed descriptor key.
// Use 'all' as a single bucket for now."
let processor = "all".to_string();
}
```
Two things are wrong with this:
1. **It's a white lie.** The function is named `stats_by_processor`, but it
returns a single `"all"` bucket. There is no per-processor grouping.
2. **It's slow.** Even to produce that single bucket, it reads and parses every
descriptor file. For 10,000 cached products that's 10,000 syscalls and
10,000 JSON parses, just to count entries.
## Why this matters: declared-but-empty processors
In `analyzers stats`, if a user declares `[analyzer.cpp]` in `rsconstruct.toml`
but the analyzer never matches anything, the table shows a `cpp 0 0` row
(implemented 2026-04-12). This is a useful signal: "you configured it, but it
is silently doing nothing."
We'd like the same in `cache stats`: show every enabled processor, including
those with zero cached entries, so that users notice mis-configurations.
We **cannot** implement this today. If we listed declared processors with
zeros, real entries would still be lumped into `"all"`, so the table would
show:
```
all: 50 entries, 58 outputs, 3.2 MiB
ruff: 0 entries, 0 outputs, 0 bytes ← misleading
pylint: 0 entries, 0 outputs, 0 bytes ← misleading
Total: 50 entries, 58 outputs, 3.2 MiB
```
That's **worse** than the current output — it tells the user "pylint produced
nothing" when pylint may actually have plenty. Fixing the 0-rows UX requires
first fixing the grouping itself.
## Options to fix per-processor cache grouping
### Option A — embed the processor name inside each descriptor
Add a `processor: String` field to `CacheDescriptor`. The cache-insert path
populates it (already known at that point). `stats_by_processor` reads the
field instead of hard-coding `"all"`.
- ✅ Small, localized change — ~100–150 lines including a backward-compat
fallback for old descriptors.
- ❌ **Does not fix the slowness.** We still read and parse every descriptor
to learn the grouping.
- ❌ Cache format change requires either a migration step, a "legacy entries
show up as `unknown`" fallback, or a cache wipe on upgrade.
### Option B — encode the processor name in the descriptor's path
Layout changes from:
```
.rsconstruct/cache/descriptors/
ab/
cd/
abcd1234…json
```
to:
```
.rsconstruct/cache/descriptors/
ruff/
abcd.json
ef01.json
pylint/
9876.json
```
`stats_by_processor` becomes:
```rust
for each subdir of descriptors/:
name = subdir.file_name() // free — already a String in the dir entry
count = number of files in subdir // one readdir per processor
```
- ✅ Fixes grouping **and** speed simultaneously. 30 `readdir`s instead of
10,000 `read`s is two to three orders of magnitude faster.
- ✅ Trivially answers "does this processor have any cached entries at all?"
with `exists(descriptors/NAME/)`.
- ❌ Changes on-disk cache layout. Requires migration.
Since descriptors are a cache by definition (regenerable from a build), the
simplest migration is: **detect the old layout on startup and wipe it.** Next
build repopulates under the new layout. No data loss beyond a slower first
build post-upgrade.
### Option C — maintain a processor→count index in a redb sidecar
Keep a small redb database (e.g. `.rsconstruct/cache/stats.redb`) with a table
mapping `processor_name → (entry_count, output_count, output_bytes)`. The
cache insert / evict paths update this index transactionally alongside the
descriptor write.
`stats_by_processor` becomes:
```rust
let db = redb::Database::open("cache/stats.redb")?;
let table = db.begin_read()?.open_table(STATS_TABLE)?;
// One DB read per processor — counts are pre-aggregated.
```
- ✅ Answers `cache stats` in O(P) where P = number of processors, independent
of cache size. Even faster than Option B at scale.
- ✅ No on-disk layout change to the descriptors themselves — the sidecar sits
alongside the existing directory structure.
- ✅ Bytes / output counts are maintained eagerly, so the "bytes" axis is also
free (unlike Option B, which still needs to `stat` each blob for bytes).
- ❌ **Two sources of truth.** If the sidecar and the descriptor directory
ever disagree (crash mid-write, manual `rm` of a descriptor, remote-cache
sync, a bug in an insert path), the UI lies. Requires either transactional
atomicity across two stores (hard — redb transaction + filesystem write) or
a periodic reconciliation pass.
- ❌ Every cache-insert path needs to update the sidecar. Miss one, and the
counts drift silently. Options B and A put the source-of-truth physically
next to the cache entry, so there's no drift to manage.
- ❌ Cache invalidation logic gets more complex: evicting a descriptor now
means "delete the file AND decrement the counter AND handle the decrement
failing." More moving parts, more places for bugs.
- ❌ Doesn't help with any future "list all entries for processor X" query —
you'd still need Option B's path layout for that, or fall back to a full
walk.
**Verdict**: Option C is the fastest for this one specific query, but it pays
for it with a consistency problem that didn't exist before. Options A and B
keep the cache self-describing — the descriptor itself (or its path) IS the
fact — so they're immune to drift.
### Option comparison
| Grouping correctness | yes | yes | yes (if kept in sync) |
| Scan cost | O(N) reads | O(P) readdirs | O(P) DB reads |
| Bytes count free | no | no (still stat blobs) | yes (pre-aggregated) |
| On-disk layout change | descriptor format | directory layout | new sidecar file |
| Source of truth | descriptor | descriptor path | **two stores** |
| Drift risk | none | none | real — needs reconciliation |
| Migration cost | wipe or dual-read | wipe | initial scan to populate |
| Code complexity | low | low | medium-high |
| Helps other queries | no | yes (list-by-processor) | no |
### Recommendation
**Option B.** The extra invasiveness is one-time (migration). The speed and
correctness wins are permanent; the path layout is self-describing, so no
drift risk; and it also unlocks fast "list entries for processor X" queries
that Options A and C don't.
Option C is attractive if the only query we cared about was a single summary,
but the sidecar's consistency burden is real and tends to surface as bugs in
edge cases (remote-cache sync, partial writes, manual cleanup).
### On Option B's "cost"
The only new artifact on disk is **N extra directory entries at the top level
of `descriptors/`**, where N is the number of distinct processors that have
ever cached anything. In practice that's 10–30 directories. Filesystems handle
that trivially — both ext4 and btrfs are fine with thousands of top-level
entries, let alone tens.
In return we get:
- `stats_by_processor` in O(N readdirs) instead of O(cache_size reads).
- Honest "declared-but-empty" rows in `cache stats` (empty dir = 0 entries,
and there is no drift to reconcile).
- Fast "list cache entries for processor X" — a single `readdir`.
- A self-describing cache: `ls .rsconstruct/cache/descriptors/` tells you at a
glance which processors have cached anything.
The cost is negligible; the payoff is across the board.
## Implementation plan (Option B)
1. **Cache insert path.** Change the descriptor write to
`descriptors/<processor>/<hash>.json` (replacing the current
`descriptors/<hash-prefix>/<hash-suffix>/<hash>.json` sharding). The
processor name is already known at insert time — it's on the product.
2. **Cache read path.** Descriptor lookups happen by cache key. If the lookup
caller already has the processor name, read directly. Otherwise scan the
processor subdirs (rare path — most lookups come from a build graph where
the processor is known).
3. **`stats_by_processor` rewrite.** Iterate subdirs of `descriptors/`; each
subdir name is a processor. Count files within. For the "bytes" axis,
continue to stat the corresponding blob objects.
4. **Migration.** On startup, if old-layout descriptor files exist (files
directly under sharded `ab/cd/` subdirs, or anywhere that isn't a
recognized processor name), wipe `descriptors/`. Cache is regenerable by
definition; next build repopulates under the new layout. Users pay one
slower build post-upgrade, no data loss.
5. **`cache stats` UX.** Once grouping is real, enumerate declared processors
from `rsconstruct.toml` and union them with processors present in
`descriptors/`. Show a 0-row for anything declared-but-empty (mirrors the
`analyzers stats` treatment already implemented in `builder/analyzers.rs`).
### Scope
Most of the work lives in `src/object_store/`:
- `management.rs` — `stats_by_processor` rewrite.
- The insert/read paths (split across `object_store.rs` and neighbors) —
path-construction change.
- The cache-clean / trim paths — updated to walk the new layout.
Followed by a small change in `src/main.rs` (`CacheAction::Stats`) to
consume the new grouped output and render a table with the declared-union
treatment.
Estimated: a couple hundred lines, concentrated in a single module.
## Secondary cleanup — graph-level helpers
Every caller that wants per-processor grouping over the current graph
currently writes the same `HashMap` pattern inline:
```rust
let mut per_processor: HashMap<&str, _> = HashMap::new();
for product in graph.products() {
per_processor.entry(&product.processor).or_default() += ...;
}
```
We could add `BuildGraph::products_by_processor() -> &HashMap<String, Vec<ProductId>>`
as a lazily-computed cached view (computed on first access, invalidated only
when the graph is mutated).
- Benefit: de-duplicates the pattern in ~5 call sites.
- Cost: caching / invalidation logic.
- Priority: **low.** The inline grouping is O(N) over RAM iteration and is
not a performance bottleneck.
Don't do this unless a sixth call site shows up.
## Current state (2026-04-12)
- `analyzers stats`: **fixed.** Shows declared-but-empty rows. Separator
between data and Total.
- `cache stats`: **unchanged.** Still uses single-bucket `"all"` grouping.
Documented as a known limitation here; fix is pending Option B.
- Graph helpers: **not added.** Inline pattern remains across call sites.
## See also
- [Cache System](cache.md) — object-store layout, descriptor keys.
- [Checksum Cache](checksum-cache.md) — mtime-based content-hash caching.
- [Dependency Caching](dependency-caching.md) — analyzer dependency cache
(which *does* have per-analyzer grouping built in).