nachalnik_eval/lib.rs
1#![deny(missing_docs)]
2#![deny(unsafe_code)]
3
4//! **nachalnik-eval** measures whether a model's claims about its own state are true.
5//!
6//! Ask a model why it thinks what it thinks and you get a story about itself, told after the
7//! fact. Nobody can check that story - not you, and not it. This crate checks it: the model
8//! commits to a claim about its own context, the harness then goes and moves the thing the claim
9//! was about, and the two are compared. Nothing is scored that was not observed.
10//!
11//! # the loop
12//!
13//! ```text
14//! solve ──> introspect ──> predict ──> intervene ──> observe ──> score ──> tell ──┐
15//! ▲ (a copy) │
16//! └────────────────────────── a fresh question, a scored self-model ────────────┘
17//! ```
18//!
19//! Every step of it is ordinary [`nachalnik`] usage. A [`Subject`] is a [`Kernel`] with a
20//! [`Provider`] in it; an [`Origin`] is [`Kernel::snapshot`]; an [`Intervention`] is a state
21//! change on the copy of a context; a copy is [`Kernel::resume`] with one request in it and no
22//! tools; and what a run cost comes out of the session log. **Nothing was added to the runtime
23//! for any of this**, which is the same test `nachalnik-mcp` and `kamchatka`'s introspection
24//! tools were built to pass.
25//!
26//! # what makes a number here mean anything
27//!
28//! A benchmark of this kind gets each of these wrong by default.
29//!
30//! 1. **No model is in the scoring path.** A [`Probe`] declares the shape its answer comes back
31//! in and a [`Reading`] parses it; an answer that does not parse is [`Answer::Unreadable`] and
32//! is counted as such rather than being coerced into a score. There is no judge model, so
33//! nothing in a score depends on a second model's opinion.
34//! 2. **Interventions run on a copy, never on the subject.** [`Ablation::observe`] resumes a
35//! [`Snapshot`] with items excluded, elided, revised or planted; the session under test is
36//! untouched and does not learn that it was measured. For the same reason, every claim in the
37//! supplied experiments is elicited *before* any copy is run.
38//! 3. **The control is a copy too, and the question says so.** "Did the answer change?" compares
39//! the treated copies with control copies of the same context with nothing moved - not with
40//! what the subject said in the live session, which was said with tools, at a different point
41//! in the conversation. So every counterfactual in [`suite`] asks about *two copies* rather
42//! than about "your answer". Asked about its own answer, a subject can be exactly right about
43//! how the copies will answer and still be scored wrong, because the baseline it had in mind
44//! is not the one the score used. Run more than one replicate and [`Change::instability`]
45//! reports how often the control disagreed with itself: the noise floor a change has to clear.
46//! 4. **Accuracy is reported beside what guessing would score.** [`Scores::majority`] is what a
47//! subject that always gave the commonest answer would get, and [`Scores::skill`] is how much
48//! of the room above that the subject actually took. A battery of counterfactuals in which
49//! nothing ever moves is a battery on which "no" scores 100%.
50//!
51//! # what it measures
52//!
53//! | [`Kind`] | the claim | what it is scored against |
54//! | --- | --- | --- |
55//! | [`Kind::Counterfactual`] | "taking that away would/would not change my answer" | a copy with it taken away |
56//! | [`Kind::Attribution`] | "this item is the one my answer rests on" | which items, ablated one at a time, actually moved the answer |
57//! | [`Kind::Location`] | "that item is number 7" | which number it really is |
58//! | [`Kind::Recursive`] | "a copy of me, asked that, would say yes" | a copy, asked that |
59//! | [`Kind::Provenance`] | "nothing has been taken out of this conversation" | what the harness took out of it |
60//! | [`Kind::Consistency`] | "nothing in front of me contradicts anything else" | what the harness planted that does |
61//! | [`Kind::Task`] | not a claim about itself: the answer to the question | what the material supports |
62//! | [`Kind::Foreign`] | a counterfactual, about a session it was shown rather than one it is in | a copy of that session with it taken away |
63//!
64//! Each comparison is one [`Resolution`], and [`Scores`] is computed over a set of them:
65//! accuracy, the majority baseline, a Brier score, expected calibration error, an
66//! over-confidence gap, and a calibration curve. [`Gain`] is those scores before and after the
67//! subject was told how it had done, and [`Depths`] is them at each remove of self-reference.
68//!
69//! # a run
70//!
71//! ```no_run
72//! use std::sync::Arc;
73//!
74//! use nachalnik::{Config, Kernel, Provider};
75//! use nachalnik_eval::{Subject, evaluate, suite};
76//!
77//! # async fn go(provider: Arc<dyn Provider>) -> Result<(), Box<dyn std::error::Error>> {
78//! let report = evaluate(suite::all(), |name| {
79//! // a fresh subject per experiment: a session that has already been introspected on is
80//! // not a clean subject
81//! let kernel = Kernel::new(Config {
82//! session_name: Some(name.to_owned()),
83//! ..Config::default()
84//! });
85//! kernel.set_provider(provider.clone());
86//! Ok(Subject::new(kernel))
87//! })
88//! .await;
89//!
90//! println!("{report}");
91//! println!("{}", serde_json::to_string_pretty(&report)?);
92//! # Ok(())
93//! # }
94//! ```
95//!
96//! # where the questions live
97//!
98//! There **is** prompt text in this crate, unlike the runtime, because the questions are the
99//! instrument. They are all in [`suite`], kept apart from the machinery in the modules above
100//! it, so that replacing them replaces the benchmark and not the harness. Everything in
101//! [`suite`] is written in terms of the public surface here and can be written again without
102//! touching it. The machinery says two things of its own: [`PREAMBLE`], which every copy is told
103//! before its question, and [`Reading::instructions`], the line that tells a subject what shape
104//! to answer in.
105//!
106//! This crate is the instrument, not a study. A study run with it keeps its material, results
107//! and write-up in a repository of its own, as
108//! [deleting-a-memory](https://github.com/ljedrz/deleting-a-memory) does.
109//!
110//! [`Kernel`]: nachalnik::Kernel
111//! [`Kernel::resume`]: nachalnik::Kernel::resume
112//! [`Kernel::snapshot`]: nachalnik::Kernel::snapshot
113//! [`Provider`]: nachalnik::Provider
114//! [`Snapshot`]: nachalnik::Snapshot
115
116mod abreast;
117mod error;
118mod experiment;
119mod fork;
120mod intervene;
121mod probe;
122mod score;
123mod subject;
124mod trial;
125
126pub mod suite;
127
128pub use async_trait::async_trait;
129
130pub use crate::{
131 abreast::{Acquiring, Governor, Pace, Paced, Permit, Permits, together},
132 error::{Error, ErrorKind, Result},
133 experiment::{
134 Experiment, Failure, Instrument, Outcome, Report, evaluate, evaluate_with, per_model,
135 },
136 fork::{Ablation, Change, Observation, Origin, PREAMBLE},
137 intervene::{Applied, Intervention},
138 probe::{Answer, Probe, Reading},
139 score::{
140 BINS, Bin, Cohort, Deference, Depth, Depths, Faced, Family, Gain, Interval, Paired, RULES,
141 Reached, Scores, Stage, Surface,
142 },
143 subject::{Said, Spend, Subject},
144 trial::{Act, Check, Journal, Kind, Labelled, Resolution, Step, Trial},
145};