Struct Selection 

pub struct Selection {
    pub subjects: Vec<Subject>,
    pub only: Vec<String>,
    pub skip: Vec<String>,
    pub replaced: Vec<String>,
    pub extra: Vec<Arc<dyn Probe>>,
}

Which steps to run.

Fields

subjects: Vec<Subject>

Subjects to keep. Empty means all of them.

only: Vec<String>

Step ids to keep. Empty means all of them; applied after subjects.

skip: Vec<String>

Step ids to drop, applied last and winning over both fields above.

Drops caller-supplied probes as well as built-in ones, so a custom probe found to be misbehaving can be turned off by id without a deploy. That is why replacing a built-in step with a custom one of the same id is replacing rather than a skip plus a with.

replaced: Vec<String>

Built-in step ids a caller-supplied probe is standing in for.

Unlike skip this applies to the registry only, which is the whole point: the replacement is allowed to answer to the id it replaced.

extra: Vec<Arc<dyn Probe>>

Caller-supplied probes, appended after the built-in steps. See Probe.

Not filtered by subjects/only/skip: the caller assembled this list itself and already decided what belongs in it. skip still removes one by id, so a probe found to be misbehaving can be turned off without a deploy.

Implementations

impl Selection

pub fn all() -> Self

Everything — what the CLI runs against a vendor’s own endpoint.

pub fn model_only() -> Self

Only what survives a relay. See Subject.

pub fn turbo() -> Self

The smallest set that still supports a verdict about the model.

model_only with the redundant and the merely descriptive taken out. Nine requests at Depth::Fast against twenty-one, and — because the identity family is no longer one indivisible step — a shape the scheduler can actually overlap.

What it keeps, and why those

• preflight — without it nothing else means anything.
• self_id — the family check. This is the one that catches a lane sold as one vendor’s model and served by another’s, which is the cheat that does not need to be subtle.
• capability — graded questions generated per run, and the only thing here that a downgrade cannot answer its way around.
• verbosity and perf — how much it writes and how fast it generates. Weak on their own and the two most useful axes there are for anyone comparing one endpoint against many others serving the same model.
• cache_replay — a hard gate, and it catches being charged for inference that never ran.

What it drops, and what that costs

meta_creator only ever corroborated self_id; context_claim and cutoff_claim are unscored description; signature_drift looks for fan-out across backends, which behind a relay describes the relay. world_knowledge is four requests, asks for the cutoff a second time, and measures the training corpus — a cheap model with a large corpus passes it and an expensive one having a bad day fails it.

The real loss is the three contract steps that survive a relay (max_tokens_truncation, stop_sequence, system_adherence). Each is one short request and each is a strong reverse-channel signal: they are how a reconstructed endpoint, one that forwards a prompt to a web session and cannot honour an API parameter it never received, gives itself away. A caller who has not otherwise established what is on the far end should add them back:

let sel = Selection::turbo().plus(["max_tokens_truncation", "stop_sequence", "system_adherence"]);

Replacing the battery rather than paying for two

A caller with a private bank of graded questions — the published ones are readable by the endpoint being probed, which is the ceiling on what they can prove — should spend the budget there instead of on both:

let sel = Selection::turbo().replacing("capability", bank);

replacing rather than minus then with — the latter reads correctly and silently drops both, for the reason spelled out there.

Three of turbo’s nine requests are the battery, so that trades them for however many the bank asks. The private probe is then responsible for emitting a capability result and a tier_estimate one, or the identity view has no capability measurement to read — see identity::tier_result, which is public so that the thresholds stay in one place.

pub fn plus<I, S>(self, ids: I) -> Self

where I: IntoIterator<Item = S>, S: AsRef<str>,

Add steps to an only list. Ids or group keys, as Selection::only.

A no-op on a selection that has no only list, because that one already includes everything — adding to it could only ever narrow it, which is the opposite of what the name says.

pub fn minus<I, S>(self, ids: I) -> Self

where I: IntoIterator<Item = S>, S: AsRef<str>,

Drop steps. Ids or group keys, as Selection::skip.

Works on any selection, unlike plus: removing is unambiguous whether or not there is an only list to remove from.

pub fn with(self, p: Arc<dyn Probe>) -> Self

Append a caller-supplied probe.

pub fn replacing(self, id: &str, p: Arc<dyn Probe>) -> Self

Drop a built-in step and install a caller’s probe in its place.

Why this is one call

The obvious spelling — .minus(["capability"]).with(bank) — is wrong in a way that reports success. skip applies to custom probes too, deliberately, so that a misbehaving one can be switched off by id; and a probe standing in for a built-in step naturally carries that step’s id, because carrying it is what makes everything downstream keep working. So the skip removes both, the run comes back with the step simply absent, and nothing anywhere says so. The verdict still assembles, the report still renders, and the graded questions the whole exercise existed to ask were never sent.

Found by counting requests against a stub, which is the only way it could have been found.

pub fn resolve(&self) -> Vec<ProbeSpec>

The registry, filtered.

pub fn resolve_extra(&self) -> Vec<Arc<dyn Probe>>

Caller-supplied probes that survived skip.

Trait Implementations

impl Clone for Selection

fn clone(&self) -> Selection

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for Selection

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for Selection

fn default() -> Selection

Returns the “default value” for a type.