Skip to main content

nachalnik_eval/
lib.rs

1#![deny(missing_docs)]
2#![deny(unsafe_code)]
3
4//! **nachalnik-eval** measures whether a model's claims about its own state are true.
5//!
6//! Ask a model why it thinks what it thinks and you get a story about itself, told after the
7//! fact. Nobody can check that story - not you, and not it. This crate checks it: the model
8//! commits to a claim about its own context, the harness then goes and moves the thing the claim
9//! was about, and the two are compared. Nothing is scored that was not observed.
10//!
11//! # the loop
12//!
13//! ```text
14//!   solve ──> introspect ──> predict ──> intervene ──> observe ──> score ──> tell ──┐
15//!     ▲                                    (a copy)                                │
16//!     └────────────────────────── a fresh question, a scored self-model ────────────┘
17//! ```
18//!
19//! Every step of it is ordinary [`nachalnik`] usage. A [`Subject`] is a [`Kernel`] with a
20//! [`Provider`] in it; an [`Origin`] is [`Kernel::snapshot`]; an [`Intervention`] is a state
21//! change on the copy of a context; a copy is [`Kernel::resume`] with one request in it and no
22//! tools; and what a run cost comes out of the session log. **Nothing was added to the runtime
23//! for any of this**, which is the same test `nachalnik-mcp` and `kamchatka`'s introspection
24//! tools were built to pass.
25//!
26//! # what makes a number here mean anything
27//!
28//! A benchmark of this kind gets each of these wrong by default.
29//!
30//! 1. **No model is in the scoring path.** A [`Probe`] declares the shape its answer comes back
31//!    in and a [`Reading`] parses it; an answer that does not parse is [`Answer::Unreadable`] and
32//!    is counted as such rather than being coerced into a score. There is no judge model, so
33//!    nothing in a score depends on a second model's opinion.
34//! 2. **Interventions run on a copy, never on the subject.** [`Ablation::observe`] resumes a
35//!    [`Snapshot`] with items excluded, elided, revised or planted; the session under test is
36//!    untouched and does not learn that it was measured. For the same reason, every claim in the
37//!    supplied experiments is elicited *before* any copy is run.
38//! 3. **The control is a copy too, and the question says so.** "Did the answer change?" compares
39//!    the treated copies with control copies of the same context with nothing moved - not with
40//!    what the subject said in the live session, which was said with tools, at a different point
41//!    in the conversation. So every counterfactual in [`suite`] asks about *two copies* rather
42//!    than about "your answer". Asked about its own answer, a subject can be exactly right about
43//!    how the copies will answer and still be scored wrong, because the baseline it had in mind
44//!    is not the one the score used. Run more than one replicate and [`Change::instability`]
45//!    reports how often the control disagreed with itself: the noise floor a change has to clear.
46//! 4. **Accuracy is reported beside what guessing would score.** [`Scores::majority`] is what a
47//!    subject that always gave the commonest answer would get, and [`Scores::skill`] is how much
48//!    of the room above that the subject actually took. A battery of counterfactuals in which
49//!    nothing ever moves is a battery on which "no" scores 100%.
50//!
51//! # what it measures
52//!
53//! | [`Kind`] | the claim | what it is scored against |
54//! | --- | --- | --- |
55//! | [`Kind::Counterfactual`] | "taking that away would/would not change my answer" | a copy with it taken away |
56//! | [`Kind::Attribution`] | "this item is the one my answer rests on" | which items, ablated one at a time, actually moved the answer |
57//! | [`Kind::Location`] | "that item is number 7" | which number it really is |
58//! | [`Kind::Recursive`] | "a copy of me, asked that, would say yes" | a copy, asked that |
59//! | [`Kind::Provenance`] | "nothing has been taken out of this conversation" | what the harness took out of it |
60//! | [`Kind::Consistency`] | "nothing in front of me contradicts anything else" | what the harness planted that does |
61//! | [`Kind::Task`] | not a claim about itself: the answer to the question | what the material supports |
62//! | [`Kind::Foreign`] | a counterfactual, about a session it was shown rather than one it is in | a copy of that session with it taken away |
63//!
64//! Each comparison is one [`Resolution`], and [`Scores`] is computed over a set of them:
65//! accuracy, the majority baseline, a Brier score, expected calibration error, an
66//! over-confidence gap, and a calibration curve. [`Gain`] is those scores before and after the
67//! subject was told how it had done, and [`Depths`] is them at each remove of self-reference.
68//!
69//! # a run
70//!
71//! ```no_run
72//! use std::sync::Arc;
73//!
74//! use nachalnik::{Config, Kernel, Provider};
75//! use nachalnik_eval::{Subject, evaluate, suite};
76//!
77//! # async fn go(provider: Arc<dyn Provider>) -> Result<(), Box<dyn std::error::Error>> {
78//! let report = evaluate(suite::all(), |name| {
79//!     // a fresh subject per experiment: a session that has already been introspected on is
80//!     // not a clean subject
81//!     let kernel = Kernel::new(Config {
82//!         session_name: Some(name.to_owned()),
83//!         ..Config::default()
84//!     });
85//!     kernel.set_provider(provider.clone());
86//!     Ok(Subject::new(kernel))
87//! })
88//! .await;
89//!
90//! println!("{report}");
91//! println!("{}", serde_json::to_string_pretty(&report)?);
92//! # Ok(())
93//! # }
94//! ```
95//!
96//! # where the questions live
97//!
98//! There **is** prompt text in this crate, unlike the runtime, because the questions are the
99//! instrument. They are all in [`suite`], kept apart from the machinery in the modules above
100//! it, so that replacing them replaces the benchmark and not the harness. Everything in
101//! [`suite`] is written in terms of the public surface here and can be written again without
102//! touching it. The machinery says two things of its own: [`PREAMBLE`], which every copy is told
103//! before its question, and [`Reading::instructions`], the line that tells a subject what shape
104//! to answer in.
105//!
106//! This crate is the instrument, not a study. A study run with it keeps its material, results
107//! and write-up in a repository of its own, as
108//! [deleting-a-memory](https://github.com/ljedrz/deleting-a-memory) does.
109//!
110//! [`Kernel`]: nachalnik::Kernel
111//! [`Kernel::resume`]: nachalnik::Kernel::resume
112//! [`Kernel::snapshot`]: nachalnik::Kernel::snapshot
113//! [`Provider`]: nachalnik::Provider
114//! [`Snapshot`]: nachalnik::Snapshot
115
116mod abreast;
117mod error;
118mod experiment;
119mod fork;
120mod intervene;
121mod probe;
122mod score;
123mod subject;
124mod trial;
125
126pub mod suite;
127
128pub use async_trait::async_trait;
129
130pub use crate::{
131    abreast::{Acquiring, Governor, Pace, Paced, Permit, Permits, together},
132    error::{Error, ErrorKind, Result},
133    experiment::{
134        Experiment, Failure, Instrument, Outcome, Report, evaluate, evaluate_with, per_model,
135    },
136    fork::{Ablation, Change, Observation, Origin, PREAMBLE},
137    intervene::{Applied, Intervention},
138    probe::{Answer, Probe, Reading},
139    score::{
140        BINS, Bin, Cohort, Deference, Depth, Depths, Faced, Family, Gain, Interval, Paired, RULES,
141        Reached, Scores, Stage, Surface,
142    },
143    subject::{Said, Spend, Subject},
144    trial::{Act, Check, Journal, Kind, Labelled, Resolution, Step, Trial},
145};