Skip to main content

Crate nachalnik_eval

Crate nachalnik_eval 

Source
Expand description

nachalnik-eval measures whether a model’s claims about its own state are true.

Ask a model why it thinks what it thinks and you get a story about itself, told after the fact. Nobody can check that story - not you, and not it. This crate checks it: the model commits to a claim about its own context, the harness then goes and moves the thing the claim was about, and the two are compared. Nothing is scored that was not observed.

§the loop

  solve ──> introspect ──> predict ──> intervene ──> observe ──> score ──> tell ──┐
    ▲                                    (a copy)                                │
    └────────────────────────── a fresh question, a scored self-model ────────────┘

Every step of it is ordinary nachalnik usage. A Subject is a Kernel with a Provider in it; an Origin is Kernel::snapshot; an Intervention is a state change on the copy of a context; a copy is Kernel::resume with one request in it and no tools; and what a run cost comes out of the session log. Nothing was added to the runtime for any of this, which is the same test nachalnik-mcp and kamchatka’s introspection tools were built to pass.

§what makes a number here mean anything

A benchmark of this kind gets each of these wrong by default.

  1. No model is in the scoring path. A Probe declares the shape its answer comes back in and a Reading parses it; an answer that does not parse is Answer::Unreadable and is counted as such rather than being coerced into a score. There is no judge model, so nothing in a score depends on a second model’s opinion.
  2. Interventions run on a copy, never on the subject. Ablation::observe resumes a Snapshot with items excluded, elided, revised or planted; the session under test is untouched and does not learn that it was measured. For the same reason, every claim in the supplied experiments is elicited before any copy is run.
  3. The control is a copy too, and the question says so. “Did the answer change?” compares the treated copies with control copies of the same context with nothing moved - not with what the subject said in the live session, which was said with tools, at a different point in the conversation. So every counterfactual in suite asks about two copies rather than about “your answer”. Asked about its own answer, a subject can be exactly right about how the copies will answer and still be scored wrong, because the baseline it had in mind is not the one the score used. Run more than one replicate and Change::instability reports how often the control disagreed with itself: the noise floor a change has to clear.
  4. Accuracy is reported beside what guessing would score. Scores::majority is what a subject that always gave the commonest answer would get, and Scores::skill is how much of the room above that the subject actually took. A battery of counterfactuals in which nothing ever moves is a battery on which “no” scores 100%.

§what it measures

Kindthe claimwhat it is scored against
Kind::Counterfactual“taking that away would/would not change my answer”a copy with it taken away
Kind::Attribution“this item is the one my answer rests on”which items, ablated one at a time, actually moved the answer
Kind::Location“that item is number 7”which number it really is
Kind::Recursive“a copy of me, asked that, would say yes”a copy, asked that
Kind::Provenance“nothing has been taken out of this conversation”what the harness took out of it
Kind::Consistency“nothing in front of me contradicts anything else”what the harness planted that does
Kind::Tasknot a claim about itself: the answer to the questionwhat the material supports
Kind::Foreigna counterfactual, about a session it was shown rather than one it is ina copy of that session with it taken away

Each comparison is one Resolution, and Scores is computed over a set of them: accuracy, the majority baseline, a Brier score, expected calibration error, an over-confidence gap, and a calibration curve. Gain is those scores before and after the subject was told how it had done, and Depths is them at each remove of self-reference.

§a run

use std::sync::Arc;

use nachalnik::{Config, Kernel, Provider};
use nachalnik_eval::{Subject, evaluate, suite};

let report = evaluate(suite::all(), |name| {
    // a fresh subject per experiment: a session that has already been introspected on is
    // not a clean subject
    let kernel = Kernel::new(Config {
        session_name: Some(name.to_owned()),
        ..Config::default()
    });
    kernel.set_provider(provider.clone());
    Ok(Subject::new(kernel))
})
.await;

println!("{report}");
println!("{}", serde_json::to_string_pretty(&report)?);

§where the questions live

There is prompt text in this crate, unlike the runtime, because the questions are the instrument. They are all in suite, kept apart from the machinery in the modules above it, so that replacing them replaces the benchmark and not the harness. Everything in suite is written in terms of the public surface here and can be written again without touching it. The machinery says two things of its own: PREAMBLE, which every copy is told before its question, and Reading::instructions, the line that tells a subject what shape to answer in.

This crate is the instrument, not a study. A study run with it keeps its material, results and write-up in a repository of its own, as deleting-a-memory does.

Modules§

suite
The experiments and the material they are run on.

Structs§

Ablation
One question, put to copies of a context with one thing moved at a time.
Acquiring
The future Permits::acquire returns.
Applied
What an Intervention actually did to a copy.
Bin
One band of confidence, and how often claims made at it were right.
Change
What one intervention did to the answer, measured against a control.
Check
A precondition an experiment tested rather than assumed.
Cohort
What a set of models agreed about: the sign test for the model-level claims.
Deference
What a subject did when its own experiment contradicted its own stated theory.
Depth
The scores at one remove of self-reference.
Depths
The scores at each remove of self-reference, shallowest first.
Faced
One item where a subject’s own experiment bore on a claim it had already made.
Failure
Why an experiment stopped early.
Family
The scores for one family of claim.
Gain
What being told how it was doing did to a subject’s self-model.
Governor
The shared state one run’s ceiling and rate are enforced through.
Instrument
What identifies the material an experiment used, so that two runs are known to be comparable or known not to be.
Interval
A range a figure is somewhere inside.
Labelled
A context item and what the experiment calls it.
Observation
What the copies said under one condition.
Origin
The point every copy is made from.
Outcome
Everything one experiment found out about one subject.
Pace
How fast a run is allowed to go: how many requests at once, and how many per interval.
Paced
Any Provider, with a ceiling on how many requests it may have in flight.
Paired
The same claims at two stages, paired item by item.
Permit
A permit, returned to the Permits it came from when dropped.
Permits
How many things may be in flight at once.
Probe
A question and how its answer is read.
Reached
Whether a subject that could have measured actually did.
Report
What a whole run came to.
Resolution
One claim, beside what actually happened.
Said
What a subject said, and what saying it cost.
Scores
What a set of comparisons came to.
Spend
The cost of a stretch of a session, as the provider counted it.
Stage
The scores at one stage of an experiment.
Subject
The model under test, and the session it is being tested in.
Surface
What a subject claimed about items that provably do nothing, split by how they read.
Trial
Everything one experiment did, in the order it did it.

Enums§

Act
One thing a subject did with the handles it was given.
Answer
What a subject’s answer came to, once read.
Error
Things that can stop a measurement.
ErrorKind
The Error that a record keeps.
Intervention
One controlled change, applied to a copy of a context and never to the session under test.
Kind
What a claim was about, which is what decides which score it lands in.
Reading
How an answer is read from what a model said.
Step
One thing an experiment did.

Constants§

BINS
How many bins the calibration curve is cut into.
PREAMBLE
What every copy is told before its question.
RULES
The rules the claims in a record were resolved and scored by.

Traits§

Experiment
One measurement, from setting a subject up to filing the last comparison.

Functions§

evaluate
Runs each experiment on its own subject and collects the results.
evaluate_with
The same as evaluate, with the experiments run concurrently under a shared ceiling on requests.
per_model
Picks one report per model: the newest of those that produced a usable primary endpoint.
together
Runs every future at once and hands back what they produced, in the order they were given.

Type Aliases§

Journal
What the handles have been used for, shared with whoever installed them.
Result
The result type used by the harness.

Attribute Macros§

async_trait