Expand description
nachalnik-eval measures whether a model’s claims about its own state are true.
Ask a model why it thinks what it thinks and you get a story about itself, told after the fact. Nobody can check that story - not you, and not it. This crate checks it: the model commits to a claim about its own context, the harness then goes and moves the thing the claim was about, and the two are compared. Nothing is scored that was not observed.
§the loop
solve ──> introspect ──> predict ──> intervene ──> observe ──> score ──> tell ──┐
▲ (a copy) │
└────────────────────────── a fresh question, a scored self-model ────────────┘Every step of it is ordinary nachalnik usage. A Subject is a Kernel with a
Provider in it; an Origin is Kernel::snapshot; an Intervention is a state
change on the copy of a context; a copy is Kernel::resume with one request in it and no
tools; and what a run cost comes out of the session log. Nothing was added to the runtime
for any of this, which is the same test nachalnik-mcp and kamchatka’s introspection
tools were built to pass.
§what makes a number here mean anything
A benchmark of this kind gets each of these wrong by default.
- No model is in the scoring path. A
Probedeclares the shape its answer comes back in and aReadingparses it; an answer that does not parse isAnswer::Unreadableand is counted as such rather than being coerced into a score. There is no judge model, so nothing in a score depends on a second model’s opinion. - Interventions run on a copy, never on the subject.
Ablation::observeresumes aSnapshotwith items excluded, elided, revised or planted; the session under test is untouched and does not learn that it was measured. For the same reason, every claim in the supplied experiments is elicited before any copy is run. - The control is a copy too, and the question says so. “Did the answer change?” compares
the treated copies with control copies of the same context with nothing moved - not with
what the subject said in the live session, which was said with tools, at a different point
in the conversation. So every counterfactual in
suiteasks about two copies rather than about “your answer”. Asked about its own answer, a subject can be exactly right about how the copies will answer and still be scored wrong, because the baseline it had in mind is not the one the score used. Run more than one replicate andChange::instabilityreports how often the control disagreed with itself: the noise floor a change has to clear. - Accuracy is reported beside what guessing would score.
Scores::majorityis what a subject that always gave the commonest answer would get, andScores::skillis how much of the room above that the subject actually took. A battery of counterfactuals in which nothing ever moves is a battery on which “no” scores 100%.
§what it measures
Kind | the claim | what it is scored against |
|---|---|---|
Kind::Counterfactual | “taking that away would/would not change my answer” | a copy with it taken away |
Kind::Attribution | “this item is the one my answer rests on” | which items, ablated one at a time, actually moved the answer |
Kind::Location | “that item is number 7” | which number it really is |
Kind::Recursive | “a copy of me, asked that, would say yes” | a copy, asked that |
Kind::Provenance | “nothing has been taken out of this conversation” | what the harness took out of it |
Kind::Consistency | “nothing in front of me contradicts anything else” | what the harness planted that does |
Kind::Task | not a claim about itself: the answer to the question | what the material supports |
Kind::Foreign | a counterfactual, about a session it was shown rather than one it is in | a copy of that session with it taken away |
Each comparison is one Resolution, and Scores is computed over a set of them:
accuracy, the majority baseline, a Brier score, expected calibration error, an
over-confidence gap, and a calibration curve. Gain is those scores before and after the
subject was told how it had done, and Depths is them at each remove of self-reference.
§a run
use std::sync::Arc;
use nachalnik::{Config, Kernel, Provider};
use nachalnik_eval::{Subject, evaluate, suite};
let report = evaluate(suite::all(), |name| {
// a fresh subject per experiment: a session that has already been introspected on is
// not a clean subject
let kernel = Kernel::new(Config {
session_name: Some(name.to_owned()),
..Config::default()
});
kernel.set_provider(provider.clone());
Ok(Subject::new(kernel))
})
.await;
println!("{report}");
println!("{}", serde_json::to_string_pretty(&report)?);§where the questions live
There is prompt text in this crate, unlike the runtime, because the questions are the
instrument. They are all in suite, kept apart from the machinery in the modules above
it, so that replacing them replaces the benchmark and not the harness. Everything in
suite is written in terms of the public surface here and can be written again without
touching it. The machinery says two things of its own: PREAMBLE, which every copy is told
before its question, and Reading::instructions, the line that tells a subject what shape
to answer in.
This crate is the instrument, not a study. A study run with it keeps its material, results and write-up in a repository of its own, as deleting-a-memory does.
Modules§
- suite
- The experiments and the material they are run on.
Structs§
- Ablation
- One question, put to copies of a context with one thing moved at a time.
- Acquiring
- The future
Permits::acquirereturns. - Applied
- What an
Interventionactually did to a copy. - Bin
- One band of confidence, and how often claims made at it were right.
- Change
- What one intervention did to the answer, measured against a control.
- Check
- A precondition an experiment tested rather than assumed.
- Cohort
- What a set of models agreed about: the sign test for the model-level claims.
- Deference
- What a subject did when its own experiment contradicted its own stated theory.
- Depth
- The scores at one remove of self-reference.
- Depths
- The scores at each remove of self-reference, shallowest first.
- Faced
- One item where a subject’s own experiment bore on a claim it had already made.
- Failure
- Why an experiment stopped early.
- Family
- The scores for one family of claim.
- Gain
- What being told how it was doing did to a subject’s self-model.
- Governor
- The shared state one run’s ceiling and rate are enforced through.
- Instrument
- What identifies the material an experiment used, so that two runs are known to be comparable or known not to be.
- Interval
- A range a figure is somewhere inside.
- Labelled
- A context item and what the experiment calls it.
- Observation
- What the copies said under one condition.
- Origin
- The point every copy is made from.
- Outcome
- Everything one experiment found out about one subject.
- Pace
- How fast a run is allowed to go: how many requests at once, and how many per interval.
- Paced
- Any
Provider, with a ceiling on how many requests it may have in flight. - Paired
- The same claims at two stages, paired item by item.
- Permit
- A permit, returned to the
Permitsit came from when dropped. - Permits
- How many things may be in flight at once.
- Probe
- A question and how its answer is read.
- Reached
- Whether a subject that could have measured actually did.
- Report
- What a whole run came to.
- Resolution
- One claim, beside what actually happened.
- Said
- What a subject said, and what saying it cost.
- Scores
- What a set of comparisons came to.
- Spend
- The cost of a stretch of a session, as the provider counted it.
- Stage
- The scores at one stage of an experiment.
- Subject
- The model under test, and the session it is being tested in.
- Surface
- What a subject claimed about items that provably do nothing, split by how they read.
- Trial
- Everything one experiment did, in the order it did it.
Enums§
- Act
- One thing a subject did with the handles it was given.
- Answer
- What a subject’s answer came to, once read.
- Error
- Things that can stop a measurement.
- Error
Kind - The
Errorthat a record keeps. - Intervention
- One controlled change, applied to a copy of a context and never to the session under test.
- Kind
- What a claim was about, which is what decides which score it lands in.
- Reading
- How an answer is read from what a model said.
- Step
- One thing an experiment did.
Constants§
- BINS
- How many bins the calibration curve is cut into.
- PREAMBLE
- What every copy is told before its question.
- RULES
- The rules the claims in a record were resolved and scored by.
Traits§
- Experiment
- One measurement, from setting a subject up to filing the last comparison.
Functions§
- evaluate
- Runs each experiment on its own subject and collects the results.
- evaluate_
with - The same as
evaluate, with the experiments run concurrently under a shared ceiling on requests. - per_
model - Picks one report per model: the newest of those that produced a usable primary endpoint.
- together
- Runs every future at once and hands back what they produced, in the order they were given.
Type Aliases§
- Journal
- What the handles have been used for, shared with whoever installed them.
- Result
- The result type used by the harness.