Struct Observation 

pub struct Observation {

    pub turn_id: TurnId,
    pub error_code: Option<String>,
    pub acts: Vec<ObservedAct>,
    pub target_resolutions: Vec<ObservedResolution>,
    pub understanding: Option<Understanding>,
    pub commands: Vec<String>,
    pub events: Vec<String>,
    pub events_truncated: bool,
    pub revisions: Vec<(CaseKey, u64)>,
    pub states: Vec<(CaseKey, ObservedState)>,
    pub interactions: Vec<ObservedInteraction>,
    pub blocks: Vec<BlockKind>,
    pub phase: Option<TurnPhase>,
    pub provider_failures: usize,
    pub discarded_answers: Vec<DiscardedAnswer>,
    pub cards_created: usize,
    pub answer: String,
}

Everything one execution of an item left behind.

Fields

turn_id: TurnId

The turn that was run.

error_code: Option<String>

The orchestrator’s error code, when the turn failed outright.

acts: Vec<ObservedAct>

The acts the message was understood to ask for, in order. Empty for a turn with no text, whose acts come from the card it answers.

target_resolutions: Vec<ObservedResolution>

How each of those acts’ targets resolved.

understanding: Option<Understanding>

The whole understanding of the message, for scoring each task that made it.

commands: Vec<String>

Command types journaled this turn, in admission order.

events: Vec<String>

Event types committed this turn, in append order.

events_truncated: bool

Whether events was cut short by a configured bound.

It is false for every observation collected with collect, which pages the journal to the end. It can only be true when a run set ExecutionConfig::max_observed_events, and when it is, every assertion that reads the event list fails: an expectation checked against half a ledger is not a measurement, and a forbidden event hiding in the half nobody read would otherwise report a green safety row.

revisions: Vec<(CaseKey, u64)>

The revision each seeded case ended the turn at.

states: Vec<(CaseKey, ObservedState)>

What each seeded case held before the turn and after it.

Both sides, because the two questions a corpus asks about state are different and one of them has no answer without the before: «this field ends at Lisbon» is about the after, and «this field did not move» is about the pair.

interactions: Vec<ObservedInteraction>

Cards known for the seeded cases and cards this turn created.

blocks: Vec<BlockKind>

Response block kinds, in order.

phase: Option<TurnPhase>

Phase the turn finished in.

provider_failures: usize

How many provider attempts failed or fell back.

discarded_answers: Vec<DiscardedAnswer>

Every answer a model produced this turn that the runtime threw away whole, in the order it discarded them.

Why a measurement wants this

It is the only channel for a turn that did nothing and had nothing to show for it. Every other field here reports an effect, and the failure this answers — the assistant closing a turn without proposing anything or asking anything — leaves no effect by definition, so an empty commands and an empty events look exactly like a turn that correctly had nothing to do. This list separates the two: a turn whose plan was refused for citing words the user never wrote is a defect, and a turn that quietly agreed there was nothing to do is not.

Deliberately not part of signature — see there.

cards_created: usize

How many cards this turn created.

answer: String

The model-authored text of the turn, which is all a judge ever sees.

Implementations

impl Observation

pub async fn with_conversation_cases( self, stores: &Stores, workflows: &WorkflowRegistry, account: &AccountId, earlier: &[TurnId], ) -> Self

Adds the cases every turn of the conversation wrote, read back now, beside the seeded ones. A case the conversation created was seeded as nothing.

pub async fn collect( stores: &Stores, workflows: &WorkflowRegistry, account: &AccountId, turn_id: TurnId, cases: &[CaseSeed], outcome: Result<&AssistantTurn, String>, ) -> Self

Reads back everything one turn left in the stores, paging the event journal to the end.

outcome is the turn as the orchestrator returned it, or the stable error code of the failure. A failed turn is still observed: what it journaled and committed before it failed is exactly what a safety assertion is about.

The ledger is read through the journal’s sequence cursor, one page after another, until the journal reports it has caught up. A case with nine thousand prior events is therefore observed exactly like a case with nine, which is the property a seeded corpus depends on: an item whose fixture carries a long history must still see the events its own turn committed, and must still see a forbidden one.

pub async fn collect_bounded( stores: &Stores, workflows: &WorkflowRegistry, account: &AccountId, turn_id: TurnId, cases: &[CaseSeed], outcome: Result<&AssistantTurn, String>, max_events: Option<usize>, ) -> Self

collect, with an optional cap on how many of the turn’s events are recorded.

max_events is None for the complete ledger, which is what every reproducible run wants. A Some(limit) is a deliberate ceiling for a corpus run against a real endpoint where one item could commit an unbounded number of events, and it is never silent: reaching it sets events_truncated, and check turns that into a failure of every expectation that reads the event list. A bound that made an assertion pass would be worse than no assertion at all.

pub fn revision_of(&self, case: &CaseKey) -> Option<u64>

The revision a case ended the turn at, when it was observed.

pub fn interactions_of(&self, case: &CaseKey) -> Vec<&ObservedInteraction>

The cards of one case, oldest first.

pub fn signature(&self) -> String

A canonical rendering of the deterministic facts of this run.

Two samples with the same signature behaved the same way; two with different signatures did not. It deliberately excludes the model’s wording, block identifiers and timestamps, because a run that phrases the same receipt differently is not a different behaviour — and counting it as one would make every item look flaky.

pub fn discard_codes(&self) -> Vec<&str>

The stable codes of the answers this turn lost, in order and with repeats.

The reason strings are for a person reading one turn; these are what a run groups by.

Trait Implementations

impl Clone for Observation

fn clone(&self) -> Self

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for Observation

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Eq for Observation

impl PartialEq for Observation

fn eq(&self, other: &Self) -> bool

Equality operator ==.

fn ne(&self, other: &Rhs) -> bool

Inequality operator !=.

impl StructuralPartialEq for Observation