Expand description
Tool-failure clustering (T0) — the flagship analyzer. Groups error Tool
grains (captured tool calls) by (tool_name, normalized error signature) and
fires when a cluster is frequent (≥ min_count) AND is either a meaningful
share of that signature’s OPPORTUNITIES (≥ min_rate) OR a large absolute
count (≥ min_abs) — so high-volume, moderate-rate failures aren’t hidden.
Emits a memory lesson. Because the signature is derived from
attacker-influenceable tool output, this analyzer never auto-applies
(§6.3) — its manifest is Never.
Opportunities, not all calls. The rate denominator is the tool’s successful calls plus this cluster — NOT every call to the tool. A call that failed at an earlier check never reached the failure being scored, so counting it as an opportunity understates every mode. Using all calls made sibling failure modes mask each other: the more distinct ways a tool broke, the smaller each mode’s share, so the tools failing in the most ways were the hardest to learn from — backwards. Measured on a real agent trace (150 tasks, 772 calls, 139 failures across 5 modes) the old denominator put every mode between 9% and 30% and the analyzer proposed nothing at all.
The failure cause (proposal row E3). With a decision backend installed
(Engine::with_decider), a cluster whose tool grains carry a cause — the
closed failure_cause vocabulary, or free text (a failure_cause string
outside it, else failure_detail) — names its majority cause
(tool_failure.cluster_cause). A known value is used as recorded. Each
distinct free-text string is classified once per run with a choice over
the vocabulary; the argmax is used when the backend is CALIBRATED and its
probability reaches CAUSE_MIN_P (the probabilities ride on the draft’s
judged_by), else the string counts as unknown. Without a backend — and
for a cluster with no cause signal at all — the draft is exactly as before.