Skip to main content

macrame/
metrics.rs

1//! What the write actor knows about its own latency (T1.4, D-079).
2//!
3//! # Why this exists
4//!
5//! [`crate::CHUNK_BUDGET`] is 3 ms and the crate has, until now, had exactly one
6//! way to find out whether that bound holds: run `benches/budgets.rs` on a
7//! synthetic fixture. That is a statement about a laptop, not about a database
8//! in use. D-059 already established that the bound does **not** hold on a large
9//! file, by a factor of 15, and it took a benchmark rewrite to notice — because
10//! nothing in the running system was counting.
11//!
12//! Tier 1's other three items are all "make the tail bounded". None of them can
13//! be validated in the field without something that measures the tail, which is
14//! why this is a precondition for them rather than a nice-to-have.
15//!
16//! # What is recorded, and what is deliberately not
17//!
18//! Four things, all of them per **actor turn** — one command, start to finish:
19//!
20//! - **queue depth** on both channels, sampled *before* the turn begins;
21//! - **hold duration**, bucketed, per command kind;
22//! - **holds over budget**, counted separately per kind;
23//! - **the longest hold since open**, with the kind that caused it.
24//!
25//! The hold is the whole turn, not the `execute` call's SQL. That is the
26//! quantity the budget is about: the SQLite write lock is not preemptible, so an
27//! interactive assertion arriving mid-turn waits for the turn, whatever the turn
28//! spent its time on.
29//!
30//! There is no per-command timestamp trail and no sampling of individual slow
31//! commands. That would be a tracing problem, and `tracing` is already a
32//! dependency — spans belong there. This module answers one question ("is the
33//! bound holding, and if not, which kind breaks it") in fixed memory, with no
34//! allocation on the actor's path.
35//!
36//! # The feature gate
37//!
38//! Behind `metrics`, which has been a **default** feature since 0.12.11
39//! (D-154): a crate whose contract is a latency bound must not ship a default
40//! build that cannot report whether the bound is met. `--no-default-features`
41//! still removes it. With the feature off, [`ActorMetrics`] is a
42//! zero-sized type whose methods compile away and [`HoldTimer::start`] does not
43//! read the clock — so the actor loop has **one** shape either way. That
44//! matters more than the nanoseconds: a `#[cfg]` in the loop body is how the
45//! instrumented and uninstrumented paths drift until only one of them is the one
46//! that runs.
47
48use std::time::Duration;
49
50/// The command kinds the actor can spend a turn on.
51///
52/// One flat enum across both channels rather than one per channel. The question
53/// this exists to answer is "which command broke the budget", and a reader
54/// looking at a 400 ms hold does not first want to know which queue it came off.
55/// Priority is a property of scheduling; kind is a property of cost.
56///
57/// # `#[non_exhaustive]`, added while it was still free (0.12.8, W4.2)
58///
59/// Adding a variant here is a **breaking change** without this attribute,
60/// because a downstream `match` on `CommandKind` would stop compiling. That is
61/// not hypothetical for this enum: [`crate::metrics::CommandKind::Rehydrate`]
62/// did not exist until 0.12.9 precisely because adding it was a break, and
63/// rehydration reported as `Archive` for several releases as a result. The
64/// codebase has already paid this cost once, which is the argument for paying
65/// the attribute now rather than deciding it at 1.0 when the cost is permanent.
66///
67/// Callers must therefore include a `_ =>` arm. In exchange, this enum can grow
68/// a variant for a command kind that does not exist yet without a major version.
69#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Hash)]
70#[repr(u8)]
71#[non_exhaustive]
72pub enum CommandKind {
73    AssertEdge,
74    RetireEdge,
75    UpsertConcept,
76    WriteBulkAtomic,
77    RebuildCurrent,
78    RegisterModel,
79    Shutdown,
80    BulkImportChunk,
81    WriteConceptsChunk,
82    WriteAnalyticsChunk,
83    UpsertEmbeddingChunk,
84    Archive,
85    RebuildFts,
86    /// The **fill** half of a chunked shadow rebuild — `Begin` and every
87    /// `Fill` chunk (T1.2).
88    ///
89    /// Its own kind rather than folded into `RebuildCurrent`, because the two
90    /// have opposite latency profiles and the whole point of the chunked path
91    /// is that its turns are short — averaging them together would hide
92    /// exactly the improvement.
93    ///
94    /// **Fill-only since 0.14.16** ([D-233]). Through 0.14.15 this kind also
95    /// carried the swap turn, which is over budget by construction, so its
96    /// `over_budget` count was `N(rebuilds) + regressions` and could not be
97    /// decomposed — the counter was a constant, not a signal. The swap is
98    /// [`CommandKind::ShadowSwap`] now, and what is left here is the half that
99    /// is *meant* to fit [`crate::CHUNK_BUDGET`]. A nonzero count on this kind
100    /// is therefore a clean canary: a fill chunk ran long, which is a
101    /// regression and nothing else.
102    ///
103    /// [D-233]: ../docs/architecture/s13-decision-register.md#d-233
104    ShadowRebuild,
105    /// Refreshing the query planner's statistics (0.12.4, D-149).
106    ///
107    /// Its own kind rather than folded into `RebuildFts`, though both are
108    /// maintenance on derived state: this one is bounded by
109    /// `PRAGMA analysis_limit` and that one is bounded by the size of the
110    /// concept table, so averaging their holds together would describe neither.
111    Analyze,
112    /// Moving archived rows back into the hot file (0.12.9, W4.3, D-152).
113    ///
114    /// Its own kind at last. Through 0.12.8 this reported as
115    /// [`CommandKind::Archive`], on the stated ground that rehydration is the
116    /// archive path run backwards and shares its budget — true of the *budget*
117    /// and false of the *attribution*, which is what a metrics surface is for.
118    /// An operator reading a long `archive` hold could not tell whether the
119    /// database had archived anything at all, and the two move rows in opposite
120    /// directions.
121    ///
122    /// The real reason it stayed folded was that adding a variant was a
123    /// breaking change. `#[non_exhaustive]` (W4.2) is what removed that
124    /// obstacle, and this variant is the first thing it bought — which is also
125    /// the evidence that the attribute was worth adding rather than a
126    /// precaution against a hypothetical.
127    ///
128    /// **Appended at the end**, per [`CommandKind::index`]: the position of
129    /// every existing variant is a persisted contract in two languages.
130    Rehydrate,
131    /// An explicit `PRAGMA wal_checkpoint` (0.12.13, W5.2, D-156).
132    ///
133    /// Its own kind because it is the one actor turn that is **not** a
134    /// transaction: it moves frames from the WAL back into the main database
135    /// file, and its duration is a function of how much WAL has accumulated
136    /// rather than of anything the caller passed. Folding it into any existing
137    /// kind would make that kind's hold distribution bimodal for a reason no
138    /// dashboard could recover.
139    ///
140    /// **Appended at the end**, per [`CommandKind::index`].
141    Checkpoint,
142    /// `PRAGMA optimize` — re-analysing only what SQLite believes has drifted
143    /// (0.13.24, W10.5, D-197).
144    ///
145    /// Split out of [`CommandKind::Analyze`], which covered both from 0.12.4 to
146    /// 0.13.23. The split is [`CommandKind::Rehydrate`]'s lesson applied before
147    /// the fact rather than after it: [D-168] refused to decide `Analyze`'s
148    /// budget exemption *because* the kind was shared, since a judgement made
149    /// about the explicit call would have landed on the automatic one —
150    /// `close()` runs `optimize()` unconditionally — without ever being made
151    /// about it.
152    ///
153    /// The two also have genuinely different hold distributions, which is the
154    /// same argument [`CommandKind::ShadowRebuild`] is separate on.
155    /// [`crate::Database::analyze`] does the work unconditionally and its hold
156    /// tracks the table. This one is a no-op when nothing has moved, so its
157    /// distribution is bimodal by design and averaging the two together
158    /// describes neither.
159    ///
160    /// **Appended at the end**, per [`CommandKind::index`].
161    ///
162    /// [D-168]: ../docs/architecture/s13-decision-register.md#d-168
163    Optimize,
164    /// Registering a lineage (0.14.7, §15.4).
165    ///
166    /// Its own kind rather than folded into `AssertEdge`, though both are one
167    /// small transaction: a fork writes to `branches` and nothing else, so its
168    /// hold is the floor an actor turn can have, and averaging it into a
169    /// command that touches four tables would flatter that command's numbers.
170    ///
171    /// Last in declaration order because that order is a persisted contract and
172    /// **new variants go at the end** — see [`CommandKind::index`]. Grouping it
173    /// next to `RegisterModel`, which is where it belongs by kind, would have
174    /// renumbered nine counters and relabelled the Python histogram's axes.
175    Fork,
176    /// Forgetting a lineage (0.14.13, §15.4, D-230).
177    ///
178    /// Its own kind rather than folded into [`CommandKind::Archive`], on
179    /// [D-152]'s finding rather than on a fresh argument: the budget really is
180    /// shared and the attribution is not, and an operator reading a long
181    /// `archive` hold could not tell whether the database had archived a
182    /// backlog of closed intervals or dropped an abandoned branch. The two also
183    /// have unrelated cost curves — one is a function of how long it has been
184    /// since the last run, the other of how much was written on one branch.
185    ///
186    /// At the end of the declaration order, per [`CommandKind::index`].
187    ///
188    /// [D-152]: ../docs/architecture/s13-decision-register.md#d-152
189    ArchiveBranch,
190    /// The **swap** turn of a chunked shadow rebuild (0.14.16, D-233).
191    ///
192    /// Split out of [`CommandKind::ShadowRebuild`], which covered both halves
193    /// from 0.6.0 to 0.14.15. This is the third instance of one shape —
194    /// [`CommandKind::Rehydrate`] out of `Archive` ([D-152]),
195    /// [`CommandKind::Optimize`] out of `Analyze` ([D-197]), this — so the
196    /// class is named where it can be seen: **one `CommandKind`, one
197    /// structural hold distribution.** A kind covering two is a defect on
198    /// arrival, to be split in review rather than found by probe.
199    ///
200    /// Here the bimodality is structural rather than workload-dependent, which
201    /// is what makes it the clearest instance of the three. Index names are
202    /// global and SQLite has no `ALTER INDEX … RENAME`, so the shadow cannot
203    /// carry `idx_lc_traversal_cover` while the live table still holds that
204    /// name — the swap is where all three indexes get built, under the write
205    /// lock. [D-082](../docs/architecture/s13-decision-register.md#d-082)
206    /// measured it at **46.8 ms**, 15.6× the budget, and it grows with the
207    /// table.
208    ///
209    /// **Exempt**, unlike its fill half — see
210    /// [`CommandKind::exempt_from_budget`], where the criterion is stated.
211    ///
212    /// At the end of the declaration order, per [`CommandKind::index`].
213    ///
214    /// [D-197]: ../docs/architecture/s13-decision-register.md#d-197
215    ShadowSwap,
216    /// Dropping a model's DiskANN index for a bulk load (0.16.2, D-276).
217    ///
218    /// Half of `bulk_embeddings`' recipe: the drop that makes a bulk load pay
219    /// for table writes only. One `DROP INDEX IF EXISTS` statement — atomic by
220    /// necessity, no smaller unit — and µs-scale in practice, but its kind
221    /// exists for attribution, not cost: a bulk load that went wrong is read
222    /// off these two counters before anything else.
223    ///
224    /// **Exempt**: one statement, no smaller unit. At the end of the
225    /// declaration order, per [`CommandKind::index`].
226    DropEmbeddingIndex,
227    /// The one-pass rebuild of a model's DiskANN index (0.16.2, D-276).
228    ///
229    /// The finish half of `bulk_embeddings`: one `CREATE INDEX` over the
230    /// table's contents, measured at **2.61 / 19.7 / 39.0 s** for 2,000
231    /// vectors at dim 64 / 256 / 512, and ~10 ms/vector at dim 256 — the
232    /// build, not the blob writes, is what a bulk embedding costs.
233    ///
234    /// **Exempt** by the same criterion as [`CommandKind::ShadowSwap`]: one
235    /// statement, atomic by necessity. The difference is that this one is
236    /// caller-scheduled, so a long hold is a choice the caller made knowing
237    /// the number — which is why the docstring on `bulk_embeddings` states it
238    /// rather than arguing it.
239    ///
240    /// At the end of the declaration order, per [`CommandKind::index`].
241    RebuildEmbeddingIndex,
242    /// Dropping or restoring the links_current mirror (0.16.3, D-277).
243    ///
244    /// The window half of `bulk_import_deferred`: the toggle that defers the
245    /// per-row projection and puts it back. One DDL statement either way —
246    /// atomic by necessity, no smaller unit — and µs-scale in practice. Its
247    /// kind exists for the same reason `DropEmbeddingIndex`'s does:
248    /// attribution beside the work it wraps, so a bulk load that went wrong
249    /// is read off these counters first.
250    ///
251    /// **Exempt**: one statement, no smaller unit. At the end of the
252    /// declaration order, per [`CommandKind::index`].
253    LinksCurrentMirror,
254}
255
256impl CommandKind {
257    /// Every kind, in declaration order. Indexing into the per-kind arrays is by
258    /// position in this slice, so the two must not drift — which is why the
259    /// arrays are sized from `ALL.len()` rather than from a hand-written count.
260    pub const ALL: &'static [CommandKind] = &[
261        CommandKind::AssertEdge,
262        CommandKind::RetireEdge,
263        CommandKind::UpsertConcept,
264        CommandKind::WriteBulkAtomic,
265        CommandKind::RebuildCurrent,
266        CommandKind::RegisterModel,
267        CommandKind::Shutdown,
268        CommandKind::BulkImportChunk,
269        CommandKind::WriteConceptsChunk,
270        CommandKind::WriteAnalyticsChunk,
271        CommandKind::UpsertEmbeddingChunk,
272        CommandKind::Archive,
273        CommandKind::RebuildFts,
274        CommandKind::ShadowRebuild,
275        CommandKind::Analyze,
276        CommandKind::Rehydrate,
277        CommandKind::Checkpoint,
278        CommandKind::Optimize,
279        CommandKind::Fork,
280        CommandKind::ArchiveBranch,
281        CommandKind::ShadowSwap,
282        CommandKind::DropEmbeddingIndex,
283        CommandKind::RebuildEmbeddingIndex,
284        CommandKind::LinksCurrentMirror,
285    ];
286
287    pub const COUNT: usize = CommandKind::ALL.len();
288
289    /// This kind's slot in the per-kind arrays.
290    ///
291    /// # Declaration order is a persisted contract (0.12.8, W4.2)
292    ///
293    /// `self as usize` means the **order of the variants above** is the order of
294    /// every per-kind array in this module, and the compiler cannot catch a
295    /// change to it. Reordering the enum silently reassigns every counter to a
296    /// different command: the code compiles, the tests pass, and a histogram
297    /// read after the change attributes `archive`'s holds to `rebuild_fts`.
298    ///
299    /// **New variants go at the end**, always — including at the end of
300    /// [`CommandKind::ALL`], whose order is what `as_str()` and the Python
301    /// surface enumerate. This binds Python too: `BUCKET_BOUNDS_MICROS` is a
302    /// module constant there and `KindMetrics` is built by position, so a
303    /// reorder here relabels axes in a language the Rust compiler is not
304    /// looking at.
305    ///
306    /// `#[repr(u8)]` is on the enum for the same reason — it pins the
307    /// discriminants to the declaration order rather than leaving them to the
308    /// compiler — but it pins them to whatever the order *is*, so it does not
309    /// make a reorder safe. Only this rule does.
310    pub const fn index(self) -> usize {
311        self as usize
312    }
313
314    pub const fn as_str(self) -> &'static str {
315        match self {
316            CommandKind::AssertEdge => "assert_edge",
317            CommandKind::RetireEdge => "retire_edge",
318            CommandKind::UpsertConcept => "upsert_concept",
319            CommandKind::WriteBulkAtomic => "write_bulk_atomic",
320            CommandKind::RebuildCurrent => "rebuild_current",
321            CommandKind::RegisterModel => "register_model",
322            CommandKind::Shutdown => "shutdown",
323            CommandKind::BulkImportChunk => "bulk_import_chunk",
324            CommandKind::WriteConceptsChunk => "write_concepts_chunk",
325            CommandKind::WriteAnalyticsChunk => "write_analytics_chunk",
326            CommandKind::UpsertEmbeddingChunk => "upsert_embedding_chunk",
327            CommandKind::Archive => "archive",
328            CommandKind::RebuildFts => "rebuild_fts",
329            CommandKind::ShadowRebuild => "shadow_rebuild",
330            CommandKind::Analyze => "analyze",
331            CommandKind::Rehydrate => "rehydrate",
332            CommandKind::Checkpoint => "checkpoint",
333            CommandKind::Optimize => "optimize",
334            CommandKind::Fork => "fork",
335            CommandKind::ArchiveBranch => "archive_branch",
336            CommandKind::ShadowSwap => "shadow_swap",
337            CommandKind::DropEmbeddingIndex => "drop_embedding_index",
338            CommandKind::RebuildEmbeddingIndex => "rebuild_embedding_index",
339            CommandKind::LinksCurrentMirror => "links_current_mirror",
340        }
341    }
342
343    /// Whether this kind is exempt from [`crate::CHUNK_BUDGET`] by contract.
344    ///
345    /// The exemptions are the table in `CHUNK_BUDGET`'s own rustdoc, and they
346    /// are carried here so a dashboard can separate "the budget is being
347    /// broken" from "the budget does not apply and never claimed to". Counting
348    /// an `archive` as a budget violation would make the violation count useless
349    /// on any database that archives.
350    ///
351    /// The two lists must agree, and since 0.12.9 they are tied together in
352    /// both directions by `the_budget_exemptions_and_their_documented_table_agree`
353    /// — the extra-row direction being the one worth having, since a table row
354    /// with no code behind it promises a caller an exemption the violation
355    /// counter is about to disagree with.
356    ///
357    /// # The criterion, stated at last (0.14.16, W12.16, [D-233])
358    ///
359    /// The register applied one rule three times without naming it, and naming
360    /// it is what let the fourth case be decided rather than argued.
361    ///
362    /// > **Exempt means the chunk bound does not apply: the operation is atomic
363    /// > by necessity and has no smaller unit. Counted means the bound applies,
364    /// > so exceeding it is information.**
365    ///
366    /// Every exemption on this list was argued that way in its own release,
367    /// whatever the summary sentence said afterwards.
368    /// [`CommandKind::WriteBulkAtomic`] ([D-014]) is one statement, and is the
369    /// kind that exists precisely because the chunked variant is *not* atomic.
370    /// [`CommandKind::Archive`] ([D-012]) and [`CommandKind::ArchiveBranch`]
371    /// delete a consistent set or none of it.
372    /// [`CommandKind::RebuildCurrent`] ([D-023]) re-derives a whole projection
373    /// in one transaction. [`CommandKind::Rehydrate`] ([D-152]) is one
374    /// unchunked transaction moving rows back across the file boundary.
375    /// [`CommandKind::Checkpoint`] ([D-156]) is a WAL boundary. None of them
376    /// has a smaller unit to chunk *into*, so 3 ms is not a bound they failed —
377    /// it is a bound that was never about them.
378    /// [`CommandKind::Analyze`] and [`CommandKind::Optimize`] ([D-197]) do have
379    /// one: they are bounded work that can take longer or shorter, so exceeding
380    /// is a fact about this database and worth counting.
381    ///
382    /// # The criterion took three tries, and the two that failed are the useful part
383    ///
384    /// **v1 — *expected-on-healthy is exempt, workload-dependent is not*.**
385    /// Falsified by reading [D-197] closely rather than by any new measurement:
386    /// `Optimize` **runs on every close** and stays counted. If expectedness
387    /// decided the question, `Optimize` would be exempt and it is not.
388    ///
389    /// **v2 — *if `over_budget` can differ from `turns`, count it*.** Falsified
390    /// by three of the exemptions themselves: an `Archive` with nothing
391    /// archivable, a `Rehydrate` of a single row and a `Checkpoint` on an empty
392    /// WAL all come in *under* budget, so their counters can differ from their
393    /// turn counts and the rule would un-exempt all three.
394    ///
395    /// Both were **observational** — read off the outcomes the existing
396    /// exemptions happened to produce, and so decidable only after the fact.
397    /// Inapplicability is decidable at design time from what the operation *is*,
398    /// which is what a criterion has to be if it is to settle the next case
399    /// rather than rationalise the last one.
400    ///
401    /// Expected-on-healthy survives as **corroboration, not definition**: a kind
402    /// with no smaller unit usually does exceed on every healthy database, so
403    /// the symptom is a fair sanity check on the diagnosis. `Optimize` is
404    /// exactly the case that shows why it cannot be the test itself.
405    ///
406    /// `over_budget` is incremented once per turn that exceeds, so it counts
407    /// **occurrences and not magnitude**. That is the fact the criterion turns
408    /// on: a kind whose every turn exceeds contributes a constant to
409    /// [`MetricsSnapshot::budget_violations`] and moves not at all when the
410    /// hold doubles. Growth is visible in this kind's histogram and
411    /// [`KindSnapshot::longest`], which no exemption touches.
412    ///
413    /// # The two halves of a shadow rebuild land on opposite sides of it
414    ///
415    /// [`CommandKind::ShadowSwap`] is exempt: ≥ 15.6× by construction ([D-082]
416    /// measured 46.8 ms against a 3 ms budget), with no healthy state in which
417    /// it fits, and routine — the crate's own end-to-end suite triggers one.
418    /// Counting it would put a permanent `N(rebuilds)` in the violation list
419    /// of every database that has ever repaired its projection, which is
420    /// [`CommandKind::Rehydrate`]'s argument exactly.
421    ///
422    /// [`CommandKind::ShadowRebuild`] — the fill half — is **not** exempt, and
423    /// that is the half [D-082] was protecting when it refused to exempt the
424    /// merged kind: *"exempting the kind would hide the first fact to excuse
425    /// the second."* The goal is reaffirmed and the mechanism superseded. The
426    /// split protects fill structurally, where non-exemption of the merged
427    /// kind only protected it in principle: a fill regression used to arrive
428    /// as `+1` on a counter that already read `N(rebuilds)`, and now it is the
429    /// only thing that can move `shadow_rebuild` off zero at all.
430    ///
431    /// `a_swap_over_budget_is_not_a_violation` is what keeps this honest, and
432    /// its fixture is the load-bearing part: it seeds enough of a graph to put
433    /// the swap **over** the budget, asserts that first, and only then asserts
434    /// the swap's own count is zero. The obvious form — run a rebuild, assert
435    /// the violation list is empty — is worthless twice over. On a small
436    /// fixture the swap finishes inside 3 ms and the assertion passes whether
437    /// the kind is exempt or not; on a real one the *fill* chunks exceed the
438    /// budget legitimately (3.14 ms at 200 keys in a debug build), so an empty
439    /// list is a property of small fixtures rather than of rebuilds.
440    ///
441    /// The two tests are **one instrument with two asymmetric halves**, and it
442    /// is worth being exact about which owns what.
443    /// `a_swap_over_budget_is_not_a_violation` owns **narrowing**: re-count the
444    /// swap and its assertion moves off zero. It cannot own widening, because
445    /// widening an exemption only ever *removes* entries from
446    /// [`MetricsSnapshot::budget_violations`] — an assertion that a count is
447    /// zero stays green under every widening, including one that swallows the
448    /// fill half whole. **Widening is owned by
449    /// `a_long_fill_is_a_violation_and_a_long_swap_is_not` below and by nothing
450    /// else**, because forging a long fill and asserting it **is** counted is
451    /// the only shape of assertion a widening can break.
452    ///
453    /// # Any claim about fill and this budget must name a build mode and a fixture size
454    ///
455    /// At fixture scale the 3 ms bound sits **inside** fill variance rather
456    /// than above it, so the same assertion is true or false depending on how
457    /// the binary was compiled and how much graph it was handed. Debug
458    /// especially: 200 keys × 4 generations puts the longest fill at 3.14 ms —
459    /// one violation, the counter working — while a release build of the same
460    /// shape stays under. A test that asserts anything about fill overages is
461    /// therefore asserting something about *its own fixture and profile*, and
462    /// has to say which. The swap is the opposite and that is why the exemption
463    /// is testable at all: it exceeds by 15.6× and it exceeds in every mode.
464    ///
465    /// [D-012]: ../docs/architecture/s13-decision-register.md#d-012
466    /// [D-014]: ../docs/architecture/s13-decision-register.md#d-014
467    /// [D-023]: ../docs/architecture/s13-decision-register.md#d-023
468    /// [D-082]: ../docs/architecture/s13-decision-register.md#d-082
469    /// [D-152]: ../docs/architecture/s13-decision-register.md#d-152
470    /// [D-156]: ../docs/architecture/s13-decision-register.md#d-156
471    /// [D-197]: ../docs/architecture/s13-decision-register.md#d-197
472    /// [D-233]: ../docs/architecture/s13-decision-register.md#d-233
473    ///
474    /// # `Rehydrate` is exempt, and splitting it out is what made that a
475    /// decision rather than an accident (0.12.9, W4.3, D-152)
476    ///
477    /// Until 0.12.8 rehydration reported as [`CommandKind::Archive`] and was
478    /// therefore exempt **by inheritance** — nobody had decided it, it fell out
479    /// of the borrowed kind. Giving it its own variant would have silently
480    /// flipped it to non-exempt, and since a rehydrate is one unchunked
481    /// transaction moving rows back across the file boundary, every single one
482    /// would have counted as a budget violation. The violation count would then
483    /// have become useless on any database that rehydrates, which is precisely
484    /// the failure the `Archive` exemption exists to prevent, arriving by the
485    /// back door of a change made for attribution.
486    ///
487    /// So it is exempt, on the merits and now on the record: rehydration is the
488    /// archive path run backwards and makes the same claim about its hold —
489    /// that it is bulk movement with no latency bound, and that the caller asked
490    /// for it explicitly.
491    ///
492    /// # Neither [`CommandKind::Analyze`] nor [`CommandKind::Optimize`] is
493    /// exempt, and since 0.13.24 those are two decisions (W10.5, D-197)
494    ///
495    /// They were one kind from 0.12.4 to 0.13.23, and [D-168] declined to decide
496    /// the exemption *because* they were: `Analyze` covered
497    /// [`crate::Database::optimize`] too, `close()` calls that unconditionally,
498    /// and so a judgement made about the explicit call would have landed on the
499    /// automatic one without ever being made about it. That is
500    /// [`CommandKind::Rehydrate`]'s lesson above arriving from the other
501    /// direction — there a shared kind *granted* an exemption nobody had
502    /// decided; here one would have *laundered* one. W10.5 split the kind so
503    /// each could be answered on its own evidence. Both answers came back the
504    /// same and the reasons are different, which is the whole reason the split
505    /// had to come first.
506    ///
507    /// **[`CommandKind::Analyze`] cannot state a `Bound`, so it cannot have a
508    /// row.** `ANALYZE` is one indivisible statement whose cost is set by data
509    /// volume — measured at **5.26 ms at 10,000 edges and 19.1 ms at 40,000**
510    /// against a 3 ms budget (`examples/analyze_hold.rs`, [D-166]). Every call
511    /// is a violation and always will be. `Checkpoint`'s bound is frames
512    /// accumulated since the last one; `Archive`'s is the session's row count.
513    /// The honest entry here would be "the size of the table, damped 3–4× by
514    /// `analysis_limit`", which is not a bound but the absence of one, and a
515    /// row that cannot fill that column is this table admitting the thing it
516    /// exists to prevent.
517    ///
518    /// **[`CommandKind::Optimize`] is not exempt for the opposite reason: its
519    /// violations are rare and they are the informative ones.** Measured
520    /// (`examples/optimize_hold.rs`, 40,000 edges): **10.7 ms the first time on
521    /// a database that has never been analysed, and 90–220 µs every time
522    /// after** — comfortably inside the budget, including immediately after a
523    /// bulk load that doubled the ledger. It is over budget only when it
524    /// actually re-analyses something, and then it is over by a lot: **460 ms**
525    /// once the table had grown 25× and SQLite's staleness ratio finally
526    /// fired. So the count is bimodal by construction and it is *reporting*
527    /// rather than complaining: an `optimize` in `budget_violations()` marks
528    /// the calls that did work, which is exactly what an operator wants to
529    /// know and exactly what exempting the kind would delete.
530    ///
531    /// **The violations are expected and must not be "fixed" by lowering
532    /// [`crate::schema::ddl::ANALYSIS_LIMIT`].** That would buy the number by
533    /// sampling too little to separate the two `source_id`-leading indices,
534    /// which is the entire purpose of having statistics ([D-149]).
535    ///
536    /// [D-149]: ../docs/architecture/s13-decision-register.md#d-149
537    /// [D-166]: ../docs/architecture/s13-decision-register.md#d-166
538    /// [D-168]: ../docs/architecture/s13-decision-register.md#d-168
539    pub const fn exempt_from_budget(self) -> bool {
540        matches!(
541            self,
542            CommandKind::WriteBulkAtomic
543                | CommandKind::Archive
544                | CommandKind::RebuildCurrent
545                | CommandKind::Rehydrate
546                | CommandKind::ArchiveBranch
547                | CommandKind::Checkpoint
548                | CommandKind::ShadowSwap
549                | CommandKind::DropEmbeddingIndex
550                | CommandKind::RebuildEmbeddingIndex
551                | CommandKind::LinksCurrentMirror
552        )
553    }
554}
555
556impl std::fmt::Display for CommandKind {
557    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
558        f.write_str(self.as_str())
559    }
560}
561
562/// Upper bounds of the hold-duration histogram, in microseconds.
563///
564/// `3_000` is [`crate::CHUNK_BUDGET`] exactly, so the bucket boundary and the
565/// bound are the same number and a reader does not have to interpolate to answer
566/// "what fraction of turns fit". The tail runs to 1 s because D-059's measured
567/// worst case was 45 ms and `rebuild_current` at 40K rows is 318 ms (D-077) —
568/// a range this has to cover without saturating.
569///
570/// Anything above the last bound lands in the overflow bucket, which is why
571/// [`KindSnapshot::buckets`] is one longer than this slice.
572pub const BUCKET_BOUNDS_MICROS: &[u64] = &[
573    100, 300, 1_000, 3_000, 10_000, 30_000, 100_000, 300_000, 1_000_000,
574];
575
576/// Number of histogram buckets, including the overflow bucket.
577pub const BUCKET_COUNT: usize = BUCKET_BOUNDS_MICROS.len() + 1;
578
579#[allow(dead_code)] // used by `imp` under `metrics`, and by the tests always
580fn bucket_of(micros: u64) -> usize {
581    // Linear scan over nine bounds. A binary search here would be slower in
582    // practice and this runs once per actor turn, against a turn measured in
583    // microseconds at best.
584    BUCKET_BOUNDS_MICROS
585        .iter()
586        .position(|&bound| micros <= bound)
587        .unwrap_or(BUCKET_BOUNDS_MICROS.len())
588}
589
590/// Times one actor turn.
591///
592/// # This clock is no longer optional (0.12.0, W1)
593///
594/// Until 0.11.0 the field was `#[cfg(feature = "metrics")]` and `elapsed()`
595/// returned `Duration::ZERO` in a default build: the reading existed only to
596/// feed [`ActorMetrics::record_hold`]'s histogram, so a build that did not keep
597/// the histogram had no reason to read a clock.
598///
599/// The chunk loop changed what the reading is *for*. A chunk's measured hold is
600/// now the input to the next chunk's size (`connection::next_chunk_size`, named
601/// in prose because it is private — D-144), which means it is a control signal
602/// in every build and not an observation in some of them. Left gated, `bulk_import` would have sized its chunks off
603/// `Duration::ZERO` — a value that reads as "comfortably under budget" — and
604/// grown every chunk to the ceiling, in exactly the builds nobody was measuring.
605///
606/// So the clock is unconditional and **only the histogram is still gated**:
607/// `record_hold` remains a no-op without the feature. What that costs is one
608/// `Instant::now()` pair per actor turn — tens of nanoseconds against a turn
609/// measured in microseconds at best, and the same reasoning §5.1.5 uses to
610/// decide that a channel hop is free beside a chunk.
611///
612/// It stays a type rather than a bare `Instant::now()` in the loop because the
613/// ordering guarantee in [`crate::connection`]'s `Turn` is attached to it.
614pub struct HoldTimer {
615    start: std::time::Instant,
616}
617
618impl HoldTimer {
619    #[inline]
620    pub fn start() -> Self {
621        Self {
622            start: std::time::Instant::now(),
623        }
624    }
625
626    #[inline]
627    pub fn elapsed(&self) -> Duration {
628        self.start.elapsed()
629    }
630}
631
632// ---------------------------------------------------------------------------
633// Instrumented implementation
634// ---------------------------------------------------------------------------
635
636#[cfg(feature = "metrics")]
637mod imp {
638    use super::{bucket_of, CommandKind, BUCKET_COUNT};
639    use std::sync::atomic::{AtomicU64, Ordering};
640    use std::time::Duration;
641
642    /// One kind's counters. All `Relaxed`: these are statistics, and ordering
643    /// them against each other would buy a consistency no reader needs and cost
644    /// fences on the write path.
645    #[derive(Debug, Default)]
646    struct Kind {
647        turns: AtomicU64,
648        total_micros: AtomicU64,
649        over_budget: AtomicU64,
650        /// This kind's own high-water mark, in µs.
651        ///
652        /// Not redundant with the global `longest`. That one names a single
653        /// command, so on any real database it names whichever kind is slowest
654        /// overall — and the question "did windowing shrink the archive's worst
655        /// hold" cannot be answered by a counter that a bulk import wins. No
656        /// packing needed here: the kind is the array index.
657        longest_micros: AtomicU64,
658        buckets: [AtomicU64; BUCKET_COUNT],
659    }
660
661    /// Live counters, shared between the actor and the handle.
662    ///
663    /// Fixed size, no allocation, no lock. The actor updates; anyone may read.
664    #[derive(Debug, Default)]
665    pub struct ActorMetrics {
666        kinds: [Kind; CommandKind::COUNT],
667        /// Packed `micros << 8 | kind`, so the longest hold and the kind that
668        /// caused it are read and written as **one** value. Two atomics would
669        /// let a reader see a duration from one turn beside a kind from
670        /// another — a rare wrong answer to exactly the question this field
671        /// exists to answer. 2^56 µs is over two thousand years.
672        ///
673        /// **The duration must occupy the high bits.** The update is a
674        /// `fetch_max` on the packed word, so whichever field is packed high is
675        /// the one being compared. The first version of this had the kind up
676        /// there, which made the "longest hold" the hold with the largest
677        /// *enum index* — a 3 ms `write_concepts_chunk` beat a 10 ms
678        /// `rebuild_current` because its variant is declared later. It was
679        /// `actor_metrics_tests` that caught it, not the unit tests, because
680        /// nothing in the arithmetic is wrong: the packing is only incorrect in
681        /// the presence of the atomic operation it exists to serve.
682        longest: AtomicU64,
683        /// Loop iterations, which is **not** the number of turns taken.
684        ///
685        /// The depth sample happens at the top of the loop, before `select!`
686        /// blocks — so an idle actor has already counted the iteration for a
687        /// command that has not arrived. That is right for depth (the sample is
688        /// "what was queued when I went looking") and wrong for turns, which is
689        /// why [`MetricsSnapshot::turns`] is the sum of the per-kind counters
690        /// instead. Conflating the two made `turns` permanently one too high and
691        /// disagree with its own breakdown.
692        depth_samples: AtomicU64,
693        high_depth_sum: AtomicU64,
694        high_depth_max: AtomicU64,
695        low_depth_sum: AtomicU64,
696        low_depth_max: AtomicU64,
697        /// Turns where the actor took high-priority work while low-priority
698        /// work was already queued (0.12.10, W4.4, D-153).
699        ///
700        /// The `biased` `select!` in `run_writer_actor` has **no floor**:
701        /// sustained high-priority traffic can hold the low tier off
702        /// indefinitely, and nothing has ever said whether that happens. This
703        /// is the numerator of that question — how often the choice went
704        /// against the low tier at all.
705        low_starved_turns: AtomicU64,
706        /// The current unbroken run of such turns. Reset to zero the moment
707        /// low-priority work is taken.
708        ///
709        /// Not exposed; it is the state [`Self::low_starved_run_max`] is a
710        /// high-water mark of. A live value would be read at an arbitrary point
711        /// in a run and mean nothing.
712        low_starved_run: AtomicU64,
713        /// The longest such run since open, which is the number that answers the
714        /// question.
715        ///
716        /// A large `low_starved_turns` on a busy database is unremarkable — it
717        /// says the high tier is being used, which is what the tier is for. A
718        /// large *run* says one specific low-priority command waited that many
719        /// turns, and it is the only one of the two that can distinguish
720        /// "prioritised" from "starved".
721        low_starved_run_max: AtomicU64,
722    }
723
724    const MICROS_SHIFT: u32 = 8;
725    const KIND_MASK: u64 = (1 << MICROS_SHIFT) - 1;
726
727    impl ActorMetrics {
728        pub fn new() -> Self {
729            Self::default()
730        }
731
732        /// Sample both queue depths. Called before the turn, not after: after
733        /// the turn the queue reflects what arrived *during* it, which is a
734        /// different and much less useful quantity.
735        #[inline]
736        pub fn record_turn(&self, high_depth: usize, low_depth: usize) {
737            self.depth_samples.fetch_add(1, Ordering::Relaxed);
738            for (sum, max, depth) in [
739                (
740                    &self.high_depth_sum,
741                    &self.high_depth_max,
742                    high_depth as u64,
743                ),
744                (&self.low_depth_sum, &self.low_depth_max, low_depth as u64),
745            ] {
746                sum.fetch_add(depth, Ordering::Relaxed);
747                max.fetch_max(depth, Ordering::Relaxed);
748            }
749        }
750
751        /// Record which tier the `select!` chose, and what was waiting.
752        ///
753        /// `low_queued` is the depth sampled *before* the `select!`, so it is
754        /// the backlog the turn found on arrival. By the time a high-priority
755        /// arm fires the low queue may have grown; using the pre-select reading
756        /// keeps this consistent with every other depth figure in this module
757        /// and makes the counter conservative — it never invents starvation
758        /// from work that arrived after the choice was made.
759        ///
760        /// A low-priority turn resets the run rather than decrementing it: the
761        /// question is "how many turns did one low-priority command wait", and
762        /// that is a run length, not a balance.
763        #[inline]
764        pub fn record_priority_choice(&self, took_high: bool, low_queued: usize) {
765            if took_high && low_queued > 0 {
766                self.low_starved_turns.fetch_add(1, Ordering::Relaxed);
767                let run = self.low_starved_run.fetch_add(1, Ordering::Relaxed) + 1;
768                self.low_starved_run_max.fetch_max(run, Ordering::Relaxed);
769            } else if !took_high {
770                self.low_starved_run.store(0, Ordering::Relaxed);
771            }
772        }
773
774        #[inline]
775        pub fn record_hold(&self, kind: CommandKind, held: Duration) {
776            let micros = held.as_micros().min(super::MICROS_CEILING as u128) as u64;
777            let k = &self.kinds[kind.index()];
778            k.turns.fetch_add(1, Ordering::Relaxed);
779            k.total_micros.fetch_add(micros, Ordering::Relaxed);
780            k.buckets[bucket_of(micros)].fetch_add(1, Ordering::Relaxed);
781            k.longest_micros.fetch_max(micros, Ordering::Relaxed);
782            if !kind.exempt_from_budget() && held > crate::CHUNK_BUDGET {
783                k.over_budget.fetch_add(1, Ordering::Relaxed);
784            }
785            self.longest.fetch_max(
786                (micros << MICROS_SHIFT) | kind.index() as u64,
787                Ordering::Relaxed,
788            );
789        }
790
791        /// A consistent-enough picture for a dashboard.
792        ///
793        /// Not a torn-read-free snapshot, and it does not pretend to be: the
794        /// actor keeps running while this walks the array, so two kinds may be
795        /// read one turn apart. Locking the actor to produce a report would make
796        /// the observer a source of the latency it is measuring.
797        pub fn snapshot(&self) -> super::MetricsSnapshot {
798            let samples = self.depth_samples.load(Ordering::Relaxed);
799            let mean = |sum: &AtomicU64| {
800                if samples == 0 {
801                    0.0
802                } else {
803                    sum.load(Ordering::Relaxed) as f64 / samples as f64
804                }
805            };
806
807            let packed = self.longest.load(Ordering::Relaxed);
808            let longest_micros = packed >> MICROS_SHIFT;
809            let longest = (longest_micros > 0)
810                .then(|| {
811                    let idx = (packed & KIND_MASK) as usize;
812                    CommandKind::ALL
813                        .get(idx)
814                        .map(|&kind| (kind, Duration::from_micros(longest_micros)))
815                })
816                .flatten();
817
818            let kinds: Vec<_> = CommandKind::ALL
819                .iter()
820                .map(|&kind| {
821                    let k = &self.kinds[kind.index()];
822                    let turns = k.turns.load(Ordering::Relaxed);
823                    let total = k.total_micros.load(Ordering::Relaxed);
824                    super::KindSnapshot {
825                        kind,
826                        turns,
827                        over_budget: k.over_budget.load(Ordering::Relaxed),
828                        mean: total
829                            .checked_div(turns)
830                            .map_or(Duration::ZERO, Duration::from_micros),
831                        longest: Duration::from_micros(k.longest_micros.load(Ordering::Relaxed)),
832                        buckets: std::array::from_fn(|i| k.buckets[i].load(Ordering::Relaxed)),
833                    }
834                })
835                .collect();
836
837            super::MetricsSnapshot {
838                // Summed, not counted separately — see `depth_samples`.
839                turns: kinds.iter().map(|k| k.turns).sum(),
840                depth_samples: samples,
841                high_depth_mean: mean(&self.high_depth_sum),
842                high_depth_max: self.high_depth_max.load(Ordering::Relaxed),
843                low_depth_mean: mean(&self.low_depth_sum),
844                low_depth_max: self.low_depth_max.load(Ordering::Relaxed),
845                low_starved_turns: self.low_starved_turns.load(Ordering::Relaxed),
846                low_starved_run_max: self.low_starved_run_max.load(Ordering::Relaxed),
847                longest,
848                kinds,
849            }
850        }
851    }
852}
853
854// ---------------------------------------------------------------------------
855// No-op implementation
856// ---------------------------------------------------------------------------
857
858#[cfg(not(feature = "metrics"))]
859mod imp {
860    use super::CommandKind;
861    use std::time::Duration;
862
863    /// The `metrics`-off shape: zero-sized, and every method is nothing.
864    #[derive(Debug, Default)]
865    pub struct ActorMetrics;
866
867    impl ActorMetrics {
868        pub fn new() -> Self {
869            Self
870        }
871        #[inline]
872        pub fn record_turn(&self, _high_depth: usize, _low_depth: usize) {}
873        #[inline]
874        pub fn record_priority_choice(&self, _took_high: bool, _low_queued: usize) {}
875        #[inline]
876        pub fn record_hold(&self, _kind: CommandKind, _held: Duration) {}
877    }
878}
879
880pub use imp::ActorMetrics;
881
882/// Saturation point for a recorded hold, in microseconds (~2,000 years).
883///
884/// Exists so the packed `longest` field cannot have a pathological duration
885/// overflow into the kind bits. A hold this long is not a measurement, it is a
886/// hang — and the counter should stay readable rather than start reporting the
887/// wrong command.
888///
889/// Kept out of the `metrics` cfg so the invariant test below runs in the default
890/// build too: the packing is a property of the layout, and a build that does not
891/// record is exactly the build where nobody would notice it break.
892#[allow(dead_code)]
893const MICROS_CEILING: u64 = (1u64 << 56) - 1;
894
895/// One command kind's holds, as of the moment [`ActorMetrics::snapshot`] read it.
896#[cfg(feature = "metrics")]
897#[derive(Debug, Clone, PartialEq, Eq)]
898#[non_exhaustive]
899pub struct KindSnapshot {
900    pub kind: CommandKind,
901    /// Turns spent on this kind.
902    pub turns: u64,
903    /// Turns that exceeded [`crate::CHUNK_BUDGET`]. Always 0 for the kinds
904    /// [`CommandKind::exempt_from_budget`] names — see there for why, and for
905    /// the criterion that decides which those are.
906    ///
907    /// **Occurrences, not magnitude**: one per turn that exceeded, however far
908    /// it exceeded by. A kind whose hold has doubled reports the same count and
909    /// a different [`Self::longest`].
910    pub over_budget: u64,
911    pub mean: Duration,
912    /// This kind's longest hold. Distinct from [`MetricsSnapshot::longest`],
913    /// which names one command across all kinds and so tends to be permanently
914    /// whichever kind is slowest overall.
915    pub longest: Duration,
916    /// Counts per [`BUCKET_BOUNDS_MICROS`], plus a final overflow bucket.
917    ///
918    /// Private behind [`Self::buckets`] since 0.12.8 (W4.2). A public array
919    /// field publishes `BUCKET_COUNT` as part of the type's shape, so adding a
920    /// bucket bound would break every caller that named the length — and the
921    /// bounds are exactly the thing a latency histogram is likely to want to
922    /// re-cut. The accessor returns a slice and the length becomes an
923    /// observation rather than a signature. Python already did it this way.
924    buckets: [u64; BUCKET_COUNT],
925}
926
927#[cfg(feature = "metrics")]
928impl KindSnapshot {
929    /// Counts per [`BUCKET_BOUNDS_MICROS`], plus a final overflow bucket.
930    ///
931    /// Pair it with `BUCKET_BOUNDS_MICROS` to label the axis rather than
932    /// hard-coding the bounds; the slice is one longer than that constant,
933    /// and the extra trailing element is the overflow bucket.
934    pub fn buckets(&self) -> &[u64] {
935        &self.buckets
936    }
937}
938
939/// What the actor has done since the database was opened.
940#[cfg(feature = "metrics")]
941#[derive(Debug, Clone, PartialEq)]
942#[non_exhaustive]
943pub struct MetricsSnapshot {
944    /// Commands executed, i.e. the sum of [`KindSnapshot::turns`]. The two agree
945    /// by construction rather than by coincidence.
946    pub turns: u64,
947    /// Loop iterations that took a queue-depth reading. Always at least
948    /// `turns + 1` on a live actor, because the reading is taken on the way in
949    /// to a `select!` that has not resolved yet. This is the denominator of the
950    /// two means below, and it is exposed so the difference is visible rather
951    /// than looking like drift.
952    pub depth_samples: u64,
953    pub high_depth_mean: f64,
954    pub high_depth_max: u64,
955    pub low_depth_mean: f64,
956    pub low_depth_max: u64,
957    /// The longest hold since open and what caused it. `None` before the first
958    /// turn, and — honestly — also when every turn so far took under a
959    /// microsecond, which on this path does not happen.
960    pub longest: Option<(CommandKind, Duration)>,
961    /// Turns spent on high-priority work while low-priority work was already
962    /// queued (0.12.10, W4.4, D-153).
963    ///
964    /// The actor's `select!` is `biased` and has **no floor**, so this is the
965    /// measurement of a bound the design has always had and never observed.
966    /// On its own it is not alarming: a busy database *should* prefer
967    /// interactive writes, and this counter rising is that working. Read it
968    /// beside [`Self::low_starved_run_max`], which is the number with teeth.
969    pub low_starved_turns: u64,
970    /// The longest unbroken run of the above — i.e. the most turns any single
971    /// low-priority command has waited (0.12.10, W4.4, D-153).
972    ///
973    /// This is the one that answers "can low-priority work be starved". A large
974    /// `low_starved_turns` spread over a long session says the tiers are doing
975    /// their job; a large *run* says one specific chunk, rebuild or archive sat
976    /// behind that many interactive writes in a row.
977    ///
978    /// # There is deliberately no forced-yield policy, and the reason changed
979    /// (0.13.26, W10.4, [D-199])
980    ///
981    /// It used to be "adding one now would be fixing a bound nobody has
982    /// observed being hit". That premise died twice. [D-153] hit the bound
983    /// completely on a synthetic burst, and W10.4 then hit it on an ordinary
984    /// one: **four closed-loop writers** — each awaiting its own write before
985    /// issuing the next, which is what application code does — starve the low
986    /// tier for essentially all of their writes
987    /// (`examples/fairness_probe.rs`). The run is bounded by how long the
988    /// caller keeps offering interactive work, not by concurrency and not by
989    /// anything in this crate.
990    ///
991    /// **What replaced it is the floor's own price.** "After N starved turns,
992    /// take one low-priority command" cannot choose *which* command — the low
993    /// queue is an mpsc channel and its head is not inspectable — and at least
994    /// one low-priority kind is exempt from [`crate::CHUNK_BUDGET`] **by
995    /// contract**: an [`crate::Database::archive`] was measured at 3.3 s
996    /// unwindowed on an 8,000-key backlog. So the floor would add an unbounded
997    /// term to the interactive worst case in order to unblock work that is
998    /// declared not to be latency-sensitive, which is the tier split running
999    /// backwards.
1000    ///
1001    /// **The lever that does work belongs to the caller**: 1 ms of think time
1002    /// between a writer's writes takes four writers from ~78 to ~2. Which makes
1003    /// this field the instrument for a decision the caller owns rather than a
1004    /// defect report about the actor.
1005    ///
1006    /// [D-153]: ../docs/architecture/s13-decision-register.md#d-153
1007    /// [D-199]: ../docs/architecture/s13-decision-register.md#d-199
1008    pub low_starved_run_max: u64,
1009    pub kinds: Vec<KindSnapshot>,
1010}
1011
1012#[cfg(feature = "metrics")]
1013impl MetricsSnapshot {
1014    /// Kinds that broke the budget, worst first. The one-line answer to "is the
1015    /// 3 ms bound holding?".
1016    pub fn budget_violations(&self) -> Vec<&KindSnapshot> {
1017        let mut v: Vec<_> = self.kinds.iter().filter(|k| k.over_budget > 0).collect();
1018        v.sort_by_key(|k| std::cmp::Reverse(k.over_budget));
1019        v
1020    }
1021
1022    /// What the budget-exempt kinds actually cost, longest first — the
1023    /// companion to [`Self::budget_violations`] and the only report that
1024    /// reaches them (0.15.28, [D-271], review A-5).
1025    ///
1026    /// # The data was always collected; the *report* dropped it
1027    ///
1028    /// A-5 reads the exemption as making these operations invisible. That is
1029    /// not quite where the gap was, and the difference decides the fix.
1030    /// `record_hold` writes turns, total, [`KindSnapshot::longest`] and the
1031    /// full histogram for **every** kind; the one thing it skips for an exempt
1032    /// one is the [`KindSnapshot::over_budget`] counter
1033    /// ([`CommandKind::exempt_from_budget`]). So the cost was measured all
1034    /// along, and then `budget_violations()` — the method every dashboard
1035    /// reaches for — filtered on `over_budget > 0`, which for an exempt kind
1036    /// is **zero by construction**. The collection path was fine and the
1037    /// reporting path threw the numbers away, permanently and silently, for
1038    /// exactly the operations whose cost nobody bounds.
1039    ///
1040    /// That set is not small: an [`crate::Database::archive`] was measured at
1041    /// 3.3 s unwindowed on an 8,000-key backlog ([D-199]), and
1042    /// `rebuild_current` at 318 ms on 40K rows ([D-077]). Those are the holds
1043    /// an operator most needs to see, and they were the ones with no way to
1044    /// be seen.
1045    ///
1046    /// # Sorted by `longest`, and that is the whole design decision
1047    ///
1048    /// [`Self::budget_violations`] sorts by `over_budget` because that is its
1049    /// severity axis. Here it is a column of zeros, so sorting by it would
1050    /// order the result by nothing at all. `longest` is the analogous
1051    /// question — *which exempt operation held the lock longest* — and it is
1052    /// the field [D-233] already established survives an exemption: `over_budget`
1053    /// counts occurrences, so a kind whose hold doubled reports the same count
1054    /// and a different `longest`. Read `longest` beside
1055    /// [`KindSnapshot::buckets`], which says whether that was the shape or one
1056    /// bad turn.
1057    ///
1058    /// # This is not a gate, and that is [D-055] rather than an omission
1059    ///
1060    /// Nothing asserts a bound on these numbers and `perf_claim_tests` gains
1061    /// no assertion here. They are *seen*, not enforced — a threshold on a
1062    /// kind that is exempt by contract would re-impose, in a test, the bound
1063    /// the exemption exists to lift.
1064    ///
1065    /// Kinds with no turns are omitted, as `budget_violations` omits kinds
1066    /// with no violations: a row of zeros for an operation the caller has
1067    /// never invoked is noise in a report meant to be read at a glance.
1068    ///
1069    /// [D-055]: ../docs/architecture/s13-decision-register.md#d-055
1070    /// [D-077]: ../docs/architecture/s13-decision-register.md#d-077
1071    /// [D-199]: ../docs/architecture/s13-decision-register.md#d-199
1072    /// [D-233]: ../docs/architecture/s13-decision-register.md#d-233
1073    /// [D-271]: ../docs/architecture/s13-decision-register.md#d-271
1074    pub fn exempt_costs(&self) -> Vec<&KindSnapshot> {
1075        let mut v: Vec<_> = self
1076            .kinds
1077            .iter()
1078            .filter(|k| k.kind.exempt_from_budget() && k.turns > 0)
1079            .collect();
1080        v.sort_by_key(|k| std::cmp::Reverse(k.longest));
1081        v
1082    }
1083}
1084
1085#[cfg(test)]
1086mod tests {
1087    use super::*;
1088
1089    #[test]
1090    fn every_kind_indexes_to_its_own_slot() {
1091        for (i, &kind) in CommandKind::ALL.iter().enumerate() {
1092            assert_eq!(kind.index(), i, "{kind} is out of order in ALL");
1093        }
1094        assert_eq!(CommandKind::COUNT, CommandKind::ALL.len());
1095    }
1096
1097    /// The budget is a bucket boundary, not a value inside one — so "fits in the
1098    /// budget" is a prefix sum and needs no interpolation.
1099    #[test]
1100    fn the_chunk_budget_is_exactly_a_bucket_boundary() {
1101        let budget = crate::CHUNK_BUDGET.as_micros() as u64;
1102        assert!(
1103            BUCKET_BOUNDS_MICROS.contains(&budget),
1104            "CHUNK_BUDGET is {budget} µs, which is not a bucket bound: \
1105             {BUCKET_BOUNDS_MICROS:?}"
1106        );
1107        assert_eq!(bucket_of(budget), bucket_of(budget - 1));
1108        assert_eq!(bucket_of(budget + 1), bucket_of(budget) + 1);
1109    }
1110
1111    #[test]
1112    fn the_overflow_bucket_catches_everything_past_the_last_bound() {
1113        let last = *BUCKET_BOUNDS_MICROS.last().unwrap();
1114        assert_eq!(bucket_of(last), BUCKET_BOUNDS_MICROS.len() - 1);
1115        assert_eq!(bucket_of(last + 1), BUCKET_COUNT - 1);
1116        assert_eq!(bucket_of(u64::MAX), BUCKET_COUNT - 1);
1117    }
1118
1119    /// The packing is the reason `longest` is one atomic: duration high, kind
1120    /// low, so a `fetch_max` on the word compares the duration.
1121    #[test]
1122    fn the_packing_leaves_room_for_both_fields() {
1123        assert!(
1124            (CommandKind::COUNT as u64) <= 0xFF,
1125            "the kind index must fit in the low 8 bits"
1126        );
1127        // The ceiling must survive being shifted up by the kind's width.
1128        assert_eq!(MICROS_CEILING.checked_shl(8), Some(MICROS_CEILING << 8));
1129        assert_eq!((MICROS_CEILING << 8) >> 8, MICROS_CEILING);
1130    }
1131
1132    /// The two halves of a shadow rebuild land on opposite sides of the
1133    /// budget, and a forged hold is the only way to assert it (0.14.16, D-233).
1134    ///
1135    /// The integration suite can run a real rebuild and check that the
1136    /// violation list comes back empty; what it cannot do is make a *fill*
1137    /// chunk run long on demand. So the canary lives here, where the hold is an
1138    /// argument: the same over-budget duration recorded against each half must
1139    /// produce a violation for one and not the other.
1140    ///
1141    /// Without this, widening the exemption to cover both halves would leave
1142    /// every test in the crate green — `a_swap_over_budget_is_not_a_violation`
1143    /// included, since it asserts a zero that a broader exemption also
1144    /// produces. This is the assertion that says the zero means *healthy* and
1145    /// not *unwatched*.
1146    #[cfg(feature = "metrics")]
1147    #[test]
1148    fn a_long_fill_is_a_violation_and_a_long_swap_is_not() {
1149        let m = ActorMetrics::new();
1150        let over = crate::CHUNK_BUDGET + Duration::from_millis(44);
1151
1152        m.record_hold(CommandKind::ShadowRebuild, over);
1153        m.record_hold(CommandKind::ShadowSwap, over);
1154
1155        let snap = m.snapshot();
1156        let of = |kind: CommandKind| {
1157            snap.kinds
1158                .iter()
1159                .find(|k| k.kind == kind)
1160                .unwrap()
1161                .over_budget
1162        };
1163
1164        assert_eq!(
1165            of(CommandKind::ShadowRebuild),
1166            1,
1167            "a fill chunk ran {over:?} against a {:?} budget and was not \
1168             counted. The fill half is the canary D-082 refused to exempt and \
1169             D-233 kept unexempted; if it stops counting, a regression on the \
1170             one path the chunked rebuild exists to keep short is invisible.",
1171            crate::CHUNK_BUDGET
1172        );
1173        assert_eq!(
1174            of(CommandKind::ShadowSwap),
1175            0,
1176            "the swap was counted as a violation. It exceeds by construction \
1177             on every healthy database, so counting it makes \
1178             `budget_violations()` nonzero forever (D-233)."
1179        );
1180
1181        // And the magnitude survives the exemption, which is the half of the
1182        // argument that decided C over B: exempting removes the *occurrence*
1183        // from the violation list and touches nothing a reader consults to see
1184        // the hold grow.
1185        let longest = snap
1186            .kinds
1187            .iter()
1188            .find(|k| k.kind == CommandKind::ShadowSwap)
1189            .unwrap()
1190            .longest;
1191        assert_eq!(
1192            longest, over,
1193            "the swap's hold stopped being recorded when it stopped being \
1194             counted. `over_budget` counts occurrences; the histogram and \
1195             `longest` are where growth is visible, and an exemption must not \
1196             reach them."
1197        );
1198    }
1199
1200    #[cfg(feature = "metrics")]
1201    #[test]
1202    fn the_longest_hold_names_the_command_that_caused_it() {
1203        let m = ActorMetrics::new();
1204        m.record_hold(CommandKind::AssertEdge, Duration::from_micros(500));
1205        m.record_hold(CommandKind::Archive, Duration::from_millis(40));
1206        m.record_hold(CommandKind::UpsertConcept, Duration::from_micros(900));
1207
1208        let snap = m.snapshot();
1209        assert_eq!(
1210            snap.longest,
1211            Some((CommandKind::Archive, Duration::from_millis(40)))
1212        );
1213    }
1214
1215    /// The regression the packing bug produced: a *short* hold of a
1216    /// later-declared kind must not outrank a long hold of an earlier one.
1217    ///
1218    /// The test above does not catch it, because `Archive` happens to be both
1219    /// the longest hold and a high enum index — which is exactly why the first
1220    /// version of the packing shipped past it. Here the two orderings disagree.
1221    #[cfg(feature = "metrics")]
1222    #[test]
1223    fn a_later_declared_kind_does_not_outrank_a_longer_hold() {
1224        let long = CommandKind::AssertEdge; // index 0
1225        let short = CommandKind::RebuildFts; // last index
1226        assert!(short.index() > long.index(), "the fixture needs the gap");
1227
1228        let m = ActorMetrics::new();
1229        m.record_hold(long, Duration::from_millis(40));
1230        m.record_hold(short, Duration::from_micros(1));
1231
1232        assert_eq!(
1233            m.snapshot().longest,
1234            Some((long, Duration::from_millis(40))),
1235            "the max is being taken over the kind index, not the duration"
1236        );
1237    }
1238
1239    /// The three contractual exemptions must not show up as violations, or the
1240    /// violation count is noise on any database that archives.
1241    #[cfg(feature = "metrics")]
1242    #[test]
1243    fn an_exempt_kind_over_budget_is_not_a_violation() {
1244        let m = ActorMetrics::new();
1245        m.record_hold(CommandKind::Archive, Duration::from_millis(40));
1246        m.record_hold(CommandKind::AssertEdge, Duration::from_millis(40));
1247
1248        let snap = m.snapshot();
1249        let violations = snap.budget_violations();
1250        assert_eq!(violations.len(), 1);
1251        assert_eq!(violations[0].kind, CommandKind::AssertEdge);
1252        assert_eq!(violations[0].over_budget, 1);
1253
1254        // But the hold is still *recorded* — exempt means "not a violation",
1255        // not "not measured". A 40 ms archive is exactly what T1.1 exists to
1256        // shrink, and it cannot be shrunk if it is not counted.
1257        let archive = snap
1258            .kinds
1259            .iter()
1260            .find(|k| k.kind == CommandKind::Archive)
1261            .unwrap();
1262        assert_eq!(archive.turns, 1);
1263        assert_eq!(archive.mean, Duration::from_millis(40));
1264    }
1265
1266    #[cfg(feature = "metrics")]
1267    #[test]
1268    fn queue_depth_is_a_mean_and_a_high_water_mark() {
1269        let m = ActorMetrics::new();
1270        m.record_turn(0, 4);
1271        m.record_turn(10, 0);
1272
1273        let snap = m.snapshot();
1274        // No command ran, so `turns` is 0 while `depth_samples` is 2. The two
1275        // counters are different facts and this is the case that shows it.
1276        assert_eq!(snap.turns, 0);
1277        assert_eq!(snap.depth_samples, 2);
1278        assert_eq!(snap.high_depth_mean, 5.0);
1279        assert_eq!(snap.high_depth_max, 10);
1280        assert_eq!(snap.low_depth_mean, 2.0);
1281        assert_eq!(snap.low_depth_max, 4);
1282    }
1283}