1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
//! # Audit Probes
//!
//! Record what a computation produced at named checkpoints, then check that
//! another run produces the same thing: on another backend, after a refactor,
//! or against a baseline stored on disk.
//!
//! A unit test asserts what a value *should* be. An audit asserts that a value
//! *has not changed*: you do not need to know the right answer, only that some
//! run you trust produced it. This makes audits a good fit for numerical
//! code whose exact outputs are hard to derive by hand, and for proving that
//! two backends agree.
//!
//! # Concepts
//!
//! * **Checkpoint**: a call such as [`AuditProbe::assert_eq_as`] in the code
//! under test. It names a value (`"x"`, `"logits"`) and says how to compare
//! it: exactly, or within a [`TolerancePolicy`] at a given float type.
//! * **Event**: what a checkpoint emits: an [`AuditProbeEventHeader`] (label,
//! source location, timestamp), [`AuditProbeEventParams`] (the comparison
//! kind, and its tolerance), and a data map of [`TensorData`]. Events are
//! passed to handlers as a borrowed [`AuditProbeEventStub`], and stored as an
//! owned [`AuditProbeEvent`].
//! * **Probe**: an [`AuditProbe`] fans each event out to a list of handlers.
//! The code under test takes a `&mut AuditProbe` and does not know what the
//! handlers do.
//! * **Handler**: an [`AuditProbeEventHandler`]. The two that matter are a
//! *recorder*, which appends each event to a stream, and a *verifier*, which
//! compares each event to the next one in an expected stream.
//! * **Stream**: the events of one run, in order. A stream can live in memory
//! ([`AuditStreamRecorder`] / [`AuditStreamVerifier`]) or on disk as a CBOR
//! [`AuditStreamFile`].
//! * **Body**: an [`AuditBody`] is the code under test, run on a device the
//! harness supplies, so a harness can run it once to record and once to
//! verify.
//! * **Baseline**: a stream stored on disk, under a location the call site
//! chooses with [`ReportsOptions`]. [`audit_baseline`] records it on the
//! first run and verifies against it after that.
//!
//! The flow is always the same: run once with a recorder, then run again with
//! a verifier fed from what was recorded.
//!
//! ```text
//! run A ── checkpoints ──▶ AuditProbe ──▶ recorder ──▶ stream ──┐
//! │ (memory or .cbor)
//! run B ── checkpoints ──▶ AuditProbe ──▶ verifier ◀────────────┘
//! │
//! └──▶ Err on the first mismatch
//! ```
//!
//! # Quick Start
//!
//! This module is built with the `audit` feature, which turns on `testing`. A
//! crate that audits in its tests enables it on its dev-dependency:
//!
//! ```toml
//! [dev-dependencies]
//! bunsen = { version = "*", features = ["audit"] }
//! ```
//!
//! ## Write a body
//!
//! A body is the code under test, run on whichever device the harness hands
//! it. A function or closure taking `(&mut AuditProbe, &Device)` is one; a
//! type implementing [`AuditBody`] is another. Put checkpoints wherever an
//! intermediate value is worth pinning down. Inputs must be the same on every
//! backend; use [`seeded_tensor`](crate::support::testing::seeded_tensor)
//! rather than `Tensor::random`.
//!
//! ```
//! # #[cfg(feature = "audit")] {
//! use bunsen::{
//! audit::AuditProbe,
//! burner::descriptors::TolerancePolicy,
//! errors::BunsenResult,
//! support::testing::seeded_tensor,
//! };
//! use burn::tensor::{
//! Device,
//! Distribution,
//! };
//!
//! fn softmax(
//! probe: &mut AuditProbe<'_>,
//! device: &Device,
//! ) -> BunsenResult<()> {
//! let x = seeded_tensor::<2>(7, [4, 8], Distribution::Default, device);
//! // Uploaded host values: bit-identical everywhere.
//! probe.assert_eq_as::<f32>("x", &x)?;
//!
//! // Computed values: close, not identical, across backends.
//! let y = burn::tensor::activation::softmax(x, 1);
//! probe.assert_approx_eq_as::<f32>(
//! "softmax(x)",
//! &y,
//! TolerancePolicy::Balanced,
//! )?;
//! Ok(())
//! }
//! # }
//! ```
//!
//! ## Compare two backends
//!
//! [`audit_across`] runs the body on the `reference` device, recording, then
//! on the `target` device, verifying. Nothing touches the disk.
//!
//! [`performance_device`](crate::support::testing::performance_device) is the
//! CPU unless a backend feature is on, so the example below compares the CPU
//! with itself in a bare `cargo test`. Run it with a backend feature (e.g.
//! `--features wgpu`) for the comparison to mean anything; see
//! [Test devices](crate::support::testing#test-devices).
//!
//! ```
//! # #[cfg(feature = "audit")] {
//! # use bunsen::{audit::AuditProbe, errors::BunsenResult};
//! # use burn::tensor::Device;
//! # fn softmax(_: &mut AuditProbe<'_>, _: &Device) -> BunsenResult<()> { Ok(()) }
//! use bunsen::{
//! audit::audit_across,
//! support::testing::{
//! cpu_device,
//! performance_device,
//! },
//! };
//!
//! audit_across(&cpu_device(), &performance_device(), &softmax).unwrap();
//! # }
//! ```
//!
//! ## Pin a baseline
//!
//! [`audit_baseline`] compares a run with a stream stored on disk. The call
//! site says where the reports live; see [Configuration](#configuration).
//!
//! ```
//! # #[cfg(feature = "audit")] {
//! # use bunsen::{audit::AuditProbe, errors::BunsenResult};
//! # use burn::tensor::Device;
//! # fn softmax(_: &mut AuditProbe<'_>, _: &Device) -> BunsenResult<()> { Ok(()) }
//! use bunsen::{
//! audit::{
//! BaselineOutcome,
//! audit_baseline,
//! reports::ReportsOptions,
//! },
//! support::testing::performance_device,
//! };
//!
//! let dir = tempfile::tempdir().unwrap();
//! let options = ReportsOptions::new(dir.path());
//! let device = performance_device();
//!
//! // No baseline yet: this run is recorded.
//! let first = audit_baseline(&options, &device, "softmax", &softmax).unwrap();
//! assert!(matches!(first, BaselineOutcome::Recorded(_)));
//!
//! // From here on, runs are verified against it.
//! let second = audit_baseline(&options, &device, "softmax", &softmax).unwrap();
//! assert!(matches!(second, BaselineOutcome::Verified(_)));
//! # }
//! ```
//!
//! ## Drive a probe by hand
//!
//! The harness functions are thin. When a test needs something else, such as
//! two different code paths on the same backend, or extra handlers, build the
//! probes yourself:
//!
//! ```
//! # #[cfg(feature = "audit")] {
//! use bunsen::{
//! audit::{
//! AuditProbe,
//! AuditStreamRecorder,
//! },
//! support::testing::cpu_device,
//! };
//! use burn::prelude::Tensor;
//!
//! let device = cpu_device();
//! let x: Tensor<1> = Tensor::arange(0..8, &device).float();
//!
//! let mut recorder = AuditStreamRecorder::default();
//! {
//! let mut probe = AuditProbe::new(vec![&mut recorder]);
//! probe.assert_eq_as::<f32>("2x", &(x.clone() * 2.0)).unwrap();
//! }
//!
//! let mut verifier = recorder.into_verifier();
//! {
//! let mut probe = AuditProbe::new(vec![&mut verifier]);
//! probe
//! .assert_eq_as::<f32>("2x", &(x.clone() + x.clone()))
//! .unwrap();
//! }
//! verifier.finish().unwrap();
//! # }
//! ```
//!
//! # Configuration
//!
//! ## Where baselines live
//!
//! The library never picks a reports location. Every call that touches the
//! disk takes it from the call site: [`audit_baseline`] and
//! [`SeriesReport::write_report`](reports::SeriesReport::write_report)
//! take a [`ReportsOptions`], and [`audit_baseline_at`] takes a path.
//!
//! A baseline named `name`, run on `device`, is stored at
//!
//! ```text
//! {root}/{backend_label}/{name}.cbor
//! ```
//!
//! where [`backend_label`](reports::backend_label) is the name of the
//! backend behind `device`, made path-safe (`cubecl<wgpu<spirv>>` becomes
//! `cubecl_wgpu_spirv`). Baselines are per backend: a CPU baseline never
//! verifies a GPU run. To compare backends with each other, use
//! [`audit_across`]. `name` may contain `/` to group baselines, but must be
//! relative and must not contain `..`.
//!
//! ## A repository default
//!
//! A repository that wants one standard location writes a constructor for its
//! default options, and its tests call that:
//!
//! ```
//! # #[cfg(feature = "audit")] {
//! use bunsen::{
//! audit::reports::{
//! BaselineMode,
//! ReportsOptions,
//! cargo_target_dir,
//! },
//! errors::BunsenResult,
//! };
//!
//! /// This repository's reports: `{target}/my-reports`; mode from the env.
//! pub fn my_reports() -> BunsenResult<ReportsOptions> {
//! let root = cargo_target_dir()
//! .unwrap_or_else(std::env::temp_dir)
//! .join("my-reports");
//! Ok(ReportsOptions::new(root)
//! .with_mode(BaselineMode::from_env("MY_REPORTS_MODE")?))
//! }
//! # }
//! ```
//!
//! [`cargo_target_dir`](reports::cargo_target_dir) is
//! `$CARGO_TARGET_DIR`, else the target directory holding the running test
//! binary, so reports land beside the build and stay out of git. To check
//! baselines *in* instead, point the root at a directory in the source tree,
//! e.g. under `env!("CARGO_MANIFEST_DIR")`.
//!
//! ## Record or verify
//!
//! [`BaselineMode`] says what [`audit_baseline`] does with a stored baseline:
//!
//! | Mode | Baseline absent | Baseline present |
//! |----------|-----------------|-------------------------|
//! | `Auto` | record | verify |
//! | `Record` | record | record (replace it) |
//! | `Verify` | error | verify |
//!
//! `Auto` is the default. Use `Record` to accept an intended change, and
//! `Verify` in CI, so that a missing baseline fails rather than silently
//! becoming the new truth. [`BaselineMode::from_env`] reads the mode from an
//! environment variable that the repository names.
//!
//! ## Choosing a checkpoint
//!
//! | Method | Compares | At |
//! |---------------------------------------|------------|-----------------------|
//! | [`assert_eq_as::<E>`] | exactly | element type `E` |
//! | [`assert_eq_cast`] | exactly | a runtime [`DType`] |
//! | [`assert_approx_eq_as::<F>`] | tolerance | float type `F` |
//! | [`assert_approx_eq_cast`] | tolerance | a runtime [`FloatDType`] |
//!
//! Each takes a label and either a `&Tensor` or a `&TensorData`; the data is
//! converted to the comparison type before it is stored. Use exact checks for
//! values that are uploaded, copied or rearranged, and approximate checks for
//! values that are computed. Across backends, float arithmetic is rarely
//! bit-identical.
//!
//! The comparison type is part of the event: comparing at `f16` deliberately
//! drops precision, and a checkpoint that changes its type no longer matches
//! its recorded event. `BF16` is compared via `f16`.
//!
//! [`assert_eq_as::<E>`]: AuditProbe::assert_eq_as
//! [`assert_eq_cast`]: AuditProbe::assert_eq_cast
//! [`assert_approx_eq_as::<F>`]: AuditProbe::assert_approx_eq_as
//! [`assert_approx_eq_cast`]: AuditProbe::assert_approx_eq_cast
//!
//! # Details
//!
//! ## How events match
//!
//! A verifier compares an incoming event with the next expected one, in this
//! order ([`try_match_events`]):
//!
//! 1. **Params** must be equal: the same comparison kind and, for approximate
//! checks, the same [`ToleranceDesc`], which holds both the float type and
//! the [`TolerancePolicy`]. Policy equality is syntactic: `Balanced` does
//! not match a `RelAbs` with the same numbers. A recorded tolerance cannot
//! be loosened silently at verify time.
//! 2. **Shapes** must be equal: the same data keys, holding the same number of
//! tensors of the same shapes.
//! 3. **Data** must match: exactly, with equal dtypes, or within the recorded
//! tolerance at the recorded float type.
//!
//! The label, source location and timestamp are **not** compared. Matching
//! goes by position in the stream, so the *n*-th checkpoint of run B is
//! checked against the *n*-th of run A, whatever it is called. The header
//! appears in the error: a mismatch carries a frame naming the event's
//! position and label, whose details hold the `file:line` of the checkpoint
//! that failed, its timestamp and its params. Keep the checkpoint sequence
//! deterministic: no checkpoints inside loops whose trip count depends on the
//! backend.
//!
//! ## Errors, not panics
//!
//! Checkpoints return [`BunsenResult`], and the first handler that fails stops
//! the event. A mismatch is a [`Policy`] error with a [`ValueMismatch`] cause.
//! The check that failed keeps its own message and evidence, and the report
//! (`{:#}`, which `ok_or_panic` panics with) prints that evidence, then the
//! event's frame. Use `?` to stop the body at the first divergence, which is
//! usually the interesting one: later checkpoints only repeat it.
//!
//! [`AuditStreamVerifier`] reports both ways a stream can be the wrong length:
//! an event past the end is an error in `on_event`, and
//! [`finish`](AuditStreamVerifier::finish) reports expected events that
//! never arrived. Call `finish` when the run ends; [`audit_across`] and
//! [`audit_baseline`] do. The older in-memory pair,
//! [`AuditProbeVecRecorder`] / [`AuditProbeVecVerifier`], panics on an event
//! past the end and has no `finish`, so it cannot detect a short run. Prefer
//! the stream handlers.
//!
//! ## Several handlers
//!
//! [`AuditProbe::new`] takes any number of handlers, and calls each in order
//! for every event. A probe can record and verify at once, for example to
//! verify against a baseline while recording a fresh stream for inspection.
//! [`AuditProbe`] is itself a handler, so probes nest.
//!
//! ## Stream files
//!
//! [`save_audit_stream`] writes an [`AuditStreamFile`] as CBOR: a format tag
//! ([`AUDIT_STREAM_FORMAT`]), a version ([`AUDIT_STREAM_VERSION`]), and the
//! events as [`AuditEventRecord`]s, the serializable mirror of
//! [`AuditProbeEvent`]. [`load_audit_stream`] rejects a file with another tag
//! or version, as [`InvalidResource`]. A missing file is a [`Lookup`]. Each
//! data map is written in key order, but timestamps are stored too, so two
//! recordings of the same run are equal as streams and not byte for byte.
//!
//! ## Custom handlers and event kinds
//!
//! A handler is any `Debug` type implementing
//! [`AuditProbeEventHandler::on_event`]; it sees each event as a borrowed
//! [`AuditProbeEventStub`] and calls
//! [`to_event`](AuditProbeEventView::to_event) to keep it. To read an
//! event's data map, use [`unpack_audit_probe_event_data!`], which unpacks one
//! or more events against a single exact pattern of keys and arities.
//!
//! ## Features
//!
//! The whole module is behind the `audit` feature. It turns on `testing`,
//! which provides the test devices and
//! [`seeded_tensor`](crate::support::testing::seeded_tensor) (and brings in
//! `rand`), and it brings in `ciborium` for the stream files.
//!
//! [`TensorData`]: burn::prelude::TensorData
//! [`DType`]: burn::tensor::DType
//! [`FloatDType`]: burn::tensor::FloatDType
//! [`TolerancePolicy`]: crate::burner::descriptors::TolerancePolicy
//! [`ToleranceDesc`]: crate::burner::descriptors::ToleranceDesc
//! [`BunsenResult`]: crate::errors::BunsenResult
//! [`Policy`]: crate::errors::BunsenErrorKind::Policy
//! [`Lookup`]: crate::errors::BunsenErrorKind::Lookup
//! [`InvalidResource`]: crate::errors::BunsenErrorKind::InvalidResource
//! [`ValueMismatch`]: crate::errors::ValueMismatch
//! [`ReportsOptions`]: reports::ReportsOptions
//! [`BaselineMode`]: reports::BaselineMode
//! [`BaselineMode::from_env`]: reports::BaselineMode::from_env
pub use *;
pub use *;
pub use *;
pub use *;
pub use *;
pub use *;
pub use *;