1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
//! Observability foundation (epic #1031, issues #1032 + #1038).
//!
//! This module is the single shared foundation every other observability issue
//! builds on. It owns OpenTelemetry initialisation, the `CQLITE_OTEL_*` config,
//! the metric-naming [`catalog`], the error-rate schema ([`ObsErrorCategory`] +
//! [`record_error`]), and graceful shutdown — and it is designed so that
//! **instrumentation call sites are identical whether or not the
//! `observability` feature is enabled.** When the feature is off, every helper
//! here compiles to a no-op and `cargo tree -p cqlite-core` links no
//! OpenTelemetry crates.
//!
//! # The contract for downstream issues (#1033–#1043)
//!
//! Downstream code MUST go through this module rather than touching
//! `opentelemetry`/`tracing-opentelemetry` directly, so telemetry stays
//! consistent and zero-cost-when-off:
//!
//! **Initialisation (CLI #1033, Flight #1041, bindings #1039/#1040):**
//! ```ignore
//! let cfg = observability::ObservabilityConfig::from_env();
//! let _guard = observability::init(cfg)?; // RAII; flushes on drop
//! // compose the tracing layer into your own subscriber (feature-gated):
//! # #[cfg(feature = "observability")]
//! tracing_subscriber::registry()
//! .with(tracing_subscriber::fmt::layer())
//! .with(observability::tracing_layer())
//! .init();
//! ```
//!
//! **Spans (read/query/write/compaction #1034–#1037):** use the ordinary
//! `tracing` macros (`#[tracing::instrument]`, `tracing::info_span!`) — they are
//! always available and the OTel layer bridges them when active. Do NOT create
//! per-cell spans on hot paths; aggregate with metrics instead.
//!
//! **Metrics:** call [`add_counter`], [`record_histogram`], [`record_gauge`]
//! with a name from [`catalog`] and bounded [`Attr`] attributes whose keys come
//! from [`catalog::attr`]:
//! ```ignore
//! use cqlite_core::observability::{self as obs, catalog, AttrValue};
//! obs::add_counter(catalog::READ_ROWS, n, &[(catalog::attr::SSTABLE_FORMAT, "bti".into())]);
//! obs::record_histogram(catalog::QUERY_DURATION, elapsed.as_secs_f64(), &[]);
//! ```
//!
//! **Errors (#1038):** at every boundary instrumented by #1034–#1037, call
//! [`record_error`] when an error escapes — it increments
//! [`catalog::ERRORS_TOTAL`] keyed by the bounded `{category, subsystem}` label
//! set and marks the active span errored. Never attach the raw error message.
//!
//! # Why was this query slow? (issue #1707)
//!
//! [`catalog::READ_DURATION`] tells you a read was slow. These four histograms tell
//! you WHERE the time went, so the next step is a decision rather than a profiler:
//!
//! 1. **Localise with `cqlite.read.phase.*`.** One sample per phase per completed
//! scan — [`catalog::READ_PHASE_IO`], [`catalog::READ_PHASE_DECOMPRESS`],
//! [`catalog::READ_PHASE_DECODE`], [`catalog::READ_PHASE_MERGE`]. Compare them
//! against EACH OTHER:
//! * **io dominant** → disk / page-cache bound (cold storage, evicted cache, a
//! network filesystem). Decode and merge tuning cannot help this read.
//! * **decompress dominant** → chunk length / compressor choice, or the same
//! chunks being decompressed repeatedly (a decompressed-chunk cache too small
//! for the scan's window).
//! * **decode dominant** → normally HEALTHY on a warm scan (decode is the CPU
//! work of a read). Suspicious only when it grows relative to the rows
//! DELIVERED: wide partitions, many collection/UDT cells, or a schema-less
//! fallback decode.
//! * **merge dominant** → too many overlapping generations, or heavy reconcile
//! (tombstones, LWW collapse). Cross-check [`catalog::COMPACTION_LAG`]; the
//! recv-wait is already excluded, so this really is merge work and not a
//! producer starving.
//! 2. **Read the ABSENCES, they are informative.** A phase that never ran records NO
//! sample rather than `0.0`: no `decompress` series means the SSTable is
//! uncompressed (the #1406 write-surface shape), and no `merge` series means the
//! read had a single generation. Absence is a fact about the read; `0.0` would be
//! a claim that a measurement was taken.
//! 3. **Know which surfaces are instrumented before you read an absence.** The
//! phases come from the STREAMING scan surfaces and the streaming
//! cross-generation merge. The materializing `SSTableManager::scan`, the BIG
//! reverse-clustering scan, the BTI trie walk, point reads and compaction reads
//! emit [`catalog::READ_DURATION`] with NO phase series at all — for those, an
//! empty breakdown means NOT MEASURED, never "fast". The full list, and why
//! instrumenting them needs async-safe propagation rather than another seam, is in
//! [`read_phase`]'s "Coverage boundary" section.
//! 4. **Check fd pressure.** [`catalog::READER_FDS_OPEN`] is what the readers hold
//! (exact, every platform, no `/proc`); [`catalog::PROC_FDS`] is the whole process
//! (sampled ~2s, Linux). A reader level climbing toward `ulimit -n` explains
//! latency that is really queueing behind failing/retried opens — and it is
//! visible BEFORE the first `EMFILE`. `PROC_FDS` minus the reader level is roughly
//! the non-reader footprint (sockets, WAL).
//! 5. **Check startup / durability stalls.** [`catalog::WAL_SIZE`] should saw-tooth;
//! a level that only climbs means flushes are not keeping up, and next open's
//! [`catalog::WAL_RECOVERY_DURATION`] grows with it. A slow FIRST query after a
//! restart is usually WAL recovery, not the read path.
//!
//! **The accounting caveat, and it matters for step 1:** the read pipeline is
//! CONCURRENT — an IO/decompress feed thread, a blocking parse thread, a merge
//! producer thread — so the phases OVERLAP in wall-clock and DO NOT sum to
//! `read.duration`. Their sum can even exceed it. They are per-phase TOTALS for
//! attribution ("which phase dominates, and how did that move between two runs?"),
//! never a decomposition of latency, and a dashboard that stacks them as a
//! breakdown of wall time will mislead. Same caveat as the #2819 `stream_*`
//! sub-phases.
//!
//! # Always-compiled vs feature-gated
//!
//! [`catalog`], [`config`], the [`ObsErrorCategory`] taxonomy, and the helper
//! signatures here are ALWAYS compiled (they pull in no OTel types), so call
//! sites and tests build in any configuration. Only the exporter/runtime wiring
//! ([`otel`]) is gated behind `observability`.
// Read-path metric emission at batch granularity (issue #1701): the accumulator
// that makes cqlite.read.{rows,bytes,partitions,duration} live instruments
// instead of documented-but-never-written ones.
pub
// Per-SCAN read-phase accumulator (issue #1707): the io/decompress/decode/merge
// buckets `ReadOpMeter` emits as `cqlite.read.phase.*` when a scan completes.
//
// `pub(crate)`, NOT `pub`: its seams live across the storage layer but entirely
// INSIDE this crate, and no downstream crate consumes any of it. That is the
// difference from `stream_subphase`, which is genuinely `pub` because
// `cqlite-flight` installs its sink. Exporting this module made ~12 items part of
// `cqlite-core`'s public API — a surface nobody asked for, that nothing in this
// repo detects a change to (#3366), and that would then have to be kept
// compatible. The ONE item a test outside the crate needs is re-exported below.
pub
pub use ;
pub use ObsErrorCategory;
// `ReadPhaseGuard` is deliberately NOT re-exported: it is only ever named as
// `install`'s return type, which callers bind with `let _g = …`.
pub use ;
/// TEST-ONLY arming surface for the read-phase io delay (issue #1707), the ONE
/// item of the crate-internal [`read_phase`] module an out-of-crate test needs.
///
/// Hidden from the rendered docs and compiled out entirely unless
/// `observability-testing` (or `cfg(test)`) is on, so a production build of
/// `cqlite-core` exports nothing here.
pub use io_delay;
pub use ;
use crate;
// Instrument construction, split from `otel.rs` per the campsite rule (#1116).
pub use ;
/// Shared in-memory OTLP capture harness for observability tests (issue #1043).
///
/// Gated behind the `observability-testing` feature, which pulls in the OTel
/// SDK's in-memory exporters (`InMemorySpanExporter` / `InMemoryMetricExporter`)
/// via `opentelemetry_sdk/testing`. Public so integration tests and future child
/// issues can reuse the same fixture API to assert span trees and metric
/// names/units/attributes. Production `observability` builds never compile this,
/// so they never link the SDK's testing surface.
/// A bounded attribute value for catalog metrics.
///
/// Mirrors the small set of types OpenTelemetry attributes accept, but is always
/// available regardless of the `observability` feature so instrumentation call
/// sites compile identically when telemetry is off. Keep values bounded
/// (prefer [`AttrValue::StaticStr`] / numbers); never feed unbounded data such
/// as raw error messages, partition keys, or full query text.
/// A single bounded metric attribute: a `&'static str` key (always from
/// [`catalog::attr`]) paired with a bounded [`AttrValue`].
pub type Attr = ;
/// Add `value` to the monotonic counter identified by a [`catalog`] name, with
/// bounded [`Attr`] attributes. No-op (and zero-cost) when the `observability`
/// feature is off.
/// Whether metrics are actually being collected — the `observability` feature is
/// compiled in AND a meter provider is installed (`otel::metrics_active`). Every
/// `record_*` here already gates on this, so it is a no-op when false; exposing it
/// lets a caller SKIP building per-request timing state whose samples would only
/// be discarded (issue #2819 M1 — the "zero-cost when the meter is off" promise).
/// Record `value` into the histogram identified by a [`catalog`] name (durations
/// in seconds, sizes in bytes — see the catalog docs). No-op when off.
/// Record `value` for the gauge identified by a [`catalog`] name (a current
/// value such as in-flight count or open SSTables). No-op when off.
/// Record an error that escaped a critical section (issue #1038).
///
/// Increments [`catalog::ERRORS_TOTAL`] keyed by the bounded
/// `{cqlite.error.category, cqlite.subsystem}` label set and marks the active
/// span as errored. `subsystem` is a `&'static str` (e.g. `"reader"`,
/// `"query"`, `"write"`, `"compaction"`) so its value space stays bounded. The
/// raw error message is never recorded. No-op when the `observability` feature
/// is off.
/// Record an error into [`catalog::ERRORS_TOTAL`] with EXTRA bounded attributes
/// beyond the canonical `{cqlite.error.category, cqlite.subsystem}` pair.
///
/// Identical to [`record_error`] but appends the caller-supplied `extra`
/// attributes to the same single counter increment (and still marks the active
/// span errored with the derived category). The extra keys MUST be bounded
/// (closed value sets), e.g. the Flight `do_get` abort taxonomy's
/// `cqlite.flight.abort_reason` (issue #2681) — never a raw message, ticket, or
/// key. No-op when the `observability` feature is off.
/// Mark the active span as errored WITHOUT counting the failure (issue #1704).
///
/// [`record_error`] does TWO things: it increments [`catalog::ERRORS_TOTAL`] and it
/// marks the active span errored. Those have different owners. The COUNT belongs to
/// the operation — exactly one per user-visible failure, which is why an inner step
/// whose caller records must not increment. The SPAN belongs to the call that opened
/// it, and an inner step that returns `Err` under an unmarked span reports a
/// successful-looking span for a failed operation.
///
/// Suppressing the whole of `record_error` therefore over-suppresses. This is the
/// span-only half, for a call site that defers only the counter. Gated identically to
/// [`record_error`] so the marked/unmarked decision cannot diverge between the two.
/// Convenience: run a `Result`-returning closure and [`record_error`] on the
/// `Err` path, returning the result unchanged. Lets call sites instrument an
/// operation without restructuring their error handling.
/// Parent a `tracing` span under a remote trace described by a W3C
/// [`traceparent`](https://www.w3.org/TR/trace-context/#traceparent-header)
/// header string (issues #1039/#1040/#1041).
///
/// Hosts (the Python/Node bindings, the Flight server) frequently receive an
/// incoming `traceparent` from a caller's tracer and want the per-call CQLite
/// span to attach to that remote trace. This helper extracts the W3C trace
/// context with the standard [`TraceContextPropagator`] and sets it as the
/// parent of `span` via `tracing-opentelemetry`'s `set_parent`.
///
/// It is always callable: when the `observability` feature is off — or when
/// `traceparent` is `None`/empty/unparseable — it is a no-op, so call sites
/// stay identical across builds. Pass the per-open or per-call traceparent
/// straight from the binding boundary; never synthesise one.
/// Inert observability guard returned by [`init`] when the `observability`
/// feature is disabled. Flushes and installs nothing.
/// Initialise observability from `cfg`. When the `observability` feature is
/// disabled this is a no-op (no OTel wiring) that returns an inert
/// [`ObservabilityGuard`], so callers (CLI, Flight, bindings) can call it
/// unconditionally.
///
/// It STILL plumbs `cfg.verify_presence_oracle` into the presence-oracle
/// false-negative verification switch (issue #2163, roborev r4): that switch is
/// an always-compiled storage-layer knob, independent of whether the OTel export
/// stack is linked, so a config-only build without the `observability` feature
/// can still enable the confirmation-scan correctness check (its counter emit is
/// simply a no-op in that build, per the module's zero-cost-when-off contract).
///
/// # Observability honesty (issue #1702, epic #1686)
///
/// When `cfg.enabled` is `true` this build CANNOT export anything, so it emits
/// ONE `WARN` naming the knob, the missing cargo feature and the consequence.
/// Without it, `CQLITE_OTEL_ENABLED=1` is a completely silent no-op and an
/// operator cannot tell "collector down / endpoint misconfigured" from "this
/// binary was built without the feature". It stays a warning, never an error:
/// degraded-but-running is the correct behavior — the defect was VISIBILITY.