quietset
English | 日本語
A model-agnostic stability filter — keeps samples whose labels or scores remain consistent across evaluators, budgets, seeds, and model checkpoints.
quietset is not a model trainer, annotation platform, or image-quality auditor. It is a small stability-filtering primitive designed to compose with other tools.
Note: quietset measures stability, not correctness. A sample can score high because evaluators consistently agree on a wrong answer. Use
gold_label-based reliability or--decision-score lcbto add evidence-based conservatism.
Use cases
Game AI / search training data
Multiple engines, depths, or seeds evaluate the same position. Keep only positions
where the evaluation is stable — consistent labels and scores regardless of search parameters.
--profile game-ai already weights signed-score agreement and defaults --decision-score
to lcb, so a small centipawn-like magnitude near zero can't masquerade as "stable" when
its sign is actually flipping under search noise (see stability_score below for why raw
stability alone isn't a keep/drop guarantee). Evaluating with several distinct engines?
Use --profile game-ai. Evaluating with one engine at multiple search depths instead? Use
--profile game-ai-single-engine — same weights, but without a min-evaluators floor that a
single-engine setup could never satisfy.
# Spend additional search budget only on positions worth re-evaluating, not the whole set.
# Gate a training run on held-out precision before trusting the kept set.
LLM judge pipelines
Multiple judge models or prompts evaluate the same response. Keep only responses where judges consistently agree, using Wilson LCB to guard against low-n flukes.
Synthetic / simulation data
Scores or rewards vary across seeds, budgets, or model checkpoints. Keep samples whose quality signal is robust to these variations.
|
Installation
CLI examples
# Score observations
# Filter to stable samples
# Filter by decision
# Pipeline from stdin
|
# Aggregate statistics
# Machine-readable summary for CI
|
# Explain why a specific sample was scored the way it was
# Compare two scored files (e.g. before/after a model update)
# Per-evaluator reliability (experimental)
# CSV output
# Weight label agreement 2x, ignore score variance
# Penalise low-evidence samples: decisions use confidence-adjusted score
# Penalise low-evidence samples: Wilson LCB on label agreement (most conservative)
# Explicit --decision-score flag (preferred for scripting; --use-* are aliases)
# Apply a use-case preset (sets weight and decision-score defaults)
# Require at least 3 observations and 2 evaluators before Keep
# Filter by LCB, confidence, and dispersion
# Compare with per-component deltas (spot regressions)
# Deep diagnostic audit report
|
|
# Extract samples by diagnostic class for human review
# Get re-evaluation recommendations
# Compute risk of stably-wrong kept samples
# Compare with hypothetical policy applied to after file
# Calibrate keep_threshold from gold labels to meet a precision target
Command reference
| Command | Input | What it does |
|---|---|---|
score |
observation JSONL/CSV | Compute per-sample stability scores and decisions |
filter |
scored JSONL | Keep samples by stability, decision, LCB, confidence, or dispersion |
summary |
scored JSONL | Aggregate statistics; lcb_keep_demotions; --json for CI |
explain |
scored JSONL | Per-sample component breakdown with visual bars |
compare |
2 scored JSONL | Before/after transition matrix, regressions, component deltas, policy comparison |
reliability |
observation JSONL | Per-evaluator reliability, confusion matrix, Fleiss kappa, Krippendorff alpha |
audit |
scored JSONL | Deep diagnostic report: borderline, LCB risk, sensitivity lists |
select |
scored JSONL | Extract samples by class for human review queues (pipeable) |
recommend |
scored JSONL | Per-sample re-evaluation suggestions with reasons |
stable-wrong-risk |
observation JSONL | Rate of stably-wrong kept samples (requires gold_label) |
calibrate |
observation JSONL | Find keep_threshold meeting a precision/coverage target |
policy |
observation JSONL | Sweep keep_threshold and show the precision/coverage trade-off table |
active-review |
scored JSONL | Rank samples by re-evaluation urgency (low LCB, high entropy, dispersion, sensitivity) |
block-score |
observation JSONL | Group by block_id and classify block-level trajectory stability |
trajectory-audit |
2 observation JSONL | Diff block-level trajectory stability between a before/after checkpoint |
preflight |
observation JSONL (1 or 2) | Pre-experiment instrumentation check: field coverage, block_id uniqueness, seed counts, checkpoint correspondence |
Output formats
Output conventions differ by command — deliberately, since some commands are built for human
inspection and some for pipeline composition (see select and
--embed-stats). There is no single unified format; use this table to know
what to expect from each command before scripting against it:
| Command | Default | Flag(s) | Notes |
|---|---|---|---|
score |
JSONL | --output-format jsonl|csv |
csv is a terminal/export format — see below |
filter |
JSONL (pass-through) | none | Always echoes original input lines unchanged |
select |
JSONL (pass-through) | none | Always echoes original input lines unchanged |
reliability |
JSONL | none | One object per evaluator, plus an optional trailing kappa/alpha line |
active-review |
JSONL | none | One object per ranked sample |
block-score |
JSONL | none | One object per block |
trajectory-audit |
text | --json (JSONL, one object per block) |
Sorted most-destructive block first |
preflight |
text | --json (single pretty object) |
Exits 1 on a blocking issue (0 with --report-only); warnings never affect the exit code |
recommend |
JSONL | --text |
Only command where JSONL is the default and text is the opt-in |
stable-wrong-risk |
single pretty JSON object | none | |
calibrate |
single pretty JSON object | --output-format json|csv |
csv is a single header row + one data row |
summary |
text | --json (single pretty object) |
|
explain |
text | --json (single pretty object) |
|
compare |
text | --json (single pretty object) |
|
audit |
text | --json (single pretty object) |
|
policy |
text table | --output-format text|json|csv (legacy --json still works as an alias for json) |
json is JSONL, one line per swept threshold; csv has fixed columns (empty cells when no gold_label) |
CSV is a dead end for piping. score --output-format csv, calibrate --output-format csv,
and policy --output-format csv exist purely for spreadsheets/BI tools. No other quietset
command can parse CSV back in (filter/summary/explain/compare/audit/select/
recommend/active-review all expect JSONL StabilityReports; stable-wrong-risk/calibrate/
reliability/policy expect JSONL Observations). If you plan to pipe score's output into
another quietset command, use the default jsonl, not csv.
score also prints a kept X / Y (Z%), review N, drop M summary to stderr by default after
scoring, plus a configuration warning if applicable (see
Minimum requirements for Keep) — stdout stays pure data either
way, so this never needs --skip-invalid-style opt-in and never breaks piping; redirect stderr
(2>/dev/null) if you don't want it in scripted output.
Input JSONL format
All fields except sample_id are optional. gold_label provides the known-correct label for
a sample; when present, the reliability command uses it as ground truth instead of majority vote.
Trajectory stability fields (optional)
10 additional optional fields let a training harness feed precomputed per-run trajectory
signals into quietset for block-score/trajectory-audit and the score components below —
quietset only ever aggregates these numbers; it never reads a checkpoint or computes a gradient
itself:
| Field | Meaning |
|---|---|
shuffle_seed |
Second seed axis (data-order/shuffle), distinct from seed (init seed) |
loss_recipe |
Loss-function/training-recipe variant — same role as model_id/evaluator_id |
block_id |
Groups this sample into a training block (e.g. a 32-position block), distinct from sample_id. Must be unique within one input dataset and within one trajectory-audit before/after pair — a producer combining independent training runs into one file must namespace it itself, e.g. "{run_id}:{block_id}" |
layer_id |
Generic layer identifier for dead_unit_count/saturated_unit_count — quietset assigns no meaning to specific values (e.g. "ft", "l2" are just strings you choose) |
gradient_sign |
Signed per-run scalar; only its sign is used |
update_cosine |
Per-run cosine similarity of this update to a reference direction |
teacher_residual |
Per-run signed scalar (e.g. model output minus teacher output) |
trajectory_effect |
Signed per-run scalar driving block growth/shrink classification (e.g. a radial-norm delta); positive = growth, negative = shrink |
dead_unit_count |
Count of newly-dead units in layer_id for this (sample, run) |
saturated_unit_count |
Count of newly-saturated units in layer_id for this (sample, run) |
Split-integrity fields (optional)
2 additional optional fields identify which root artifact a sample was derived from, so
calibrate --group-by can detect when a calibration/held-out split accidentally separates
correlated samples:
| Field | Meaning |
|---|---|
source_root_id |
One game record/kifu, source document, or conversation. Samples sharing this value are not independent — nearby positions from the same game record correlate strongly. |
opening_family |
Coarser grouping above source_root_id (e.g. an opening family for game records, a topic cluster for judge data). Same independence caveat one level up. |
Output JSONL format
Key fields in the output (optional fields are omitted when not computable):
Optional fields are omitted when not computable (e.g. label_agreement_lcb only appears when
labels are present; score_mad / score_iqr require at least two numeric scores).
Stability score
The stability_score is a value in [0.0, 1.0]:
1.0= highly stable0.0= highly unstable
stability_score is a convenience aggregate for ranking and inspection — not a calibrated
probability and not a correctness guarantee. It has no uncertainty model of its own; a sample
where every evaluator consistently agrees on a wrong answer still scores 1.0. For real
accept/reject decisions, prefer a component with an actual statistical or model-based basis —
label_agreement_lcb (Wilson interval), --decision-score latent-truth (Dawid-Skene EM),
gold-weighted, or a threshold picked by calibrate --heldout — over the raw
stability_score/--keep-threshold default. See
Where these numbers come from and
docs/metrics.md for the full provenance of every field below.
It is the weighted mean of available sub-scores (all in [0.0, 1.0]):
| Component | Value |
|---|---|
label_agreement |
fraction of observations with the majority label |
score_consistency |
1 - normalized_score_std |
budget_robustness |
1 - budget_sensitivity |
seed_robustness |
1 - seed_sensitivity |
model_agreement |
label agreement across models |
evaluator_agreement |
label agreement across evaluators |
score_sign_agreement |
fraction of numeric scores sharing the majority sign — weight defaults to 0 (excluded); opt in with --weight-score-sign (--profile game-ai enables it at ×2 by default) |
gradient_sign_agreement |
fraction of gradient_sign observations sharing the majority sign — weight defaults to 0; opt in with --weight-gradient-sign |
update_direction_agreement |
fraction of update_cosine observations sharing the majority sign — weight defaults to 0; opt in with --weight-update-cosine |
teacher_residual_stability |
1 - normalized_std(teacher_residual) — weight defaults to 0; opt in with --weight-teacher-residual |
| shuffle-seed robustness | 1 - shuffle_seed_sensitivity — weight defaults to 0; opt in with --weight-shuffle-seed |
loss_recipe_agreement |
label agreement across loss_recipe values — weight defaults to 0; opt in with --weight-loss-recipe |
Missing dimensions (e.g. no labels, no budgets) are excluded from the mean.
Single observations receive stability_score = 0.5 (review by default).
Additional diagnostic fields on StabilityReport:
| Field | Meaning |
|---|---|
label_margin |
(majority_count - runner_up_count) / total. 0.0 = perfectly split |
label_entropy |
Normalised Shannon entropy [0, 1]. 1.0 = uniform label distribution |
label_agreement_lcb |
Wilson confidence interval lower bound of label_agreement. More conservative than raw label_agreement — guards against low-n coincidences. |
score_mad |
Median absolute deviation of numeric scores. More robust to outliers than score_std. |
score_iqr |
Interquartile range (Q3 − Q1) of numeric scores. |
score_sign_agreement |
Fraction of numeric scores sharing the majority sign (negative / zero / positive). Domain-aware complement to score_std/score_mad/score_iqr for signed, zero-centered scores (chess/shogi-style eval): [+0.01, -0.01] reads as near-perfectly stable by magnitude alone but is a coin-flip on sign. Always computed but excluded from stability_score by default (--weight-score-sign 0); pass --weight-score-sign <f> to make it contribute. --profile game-ai enables it at ×2 by default — for every other profile, this stays a purely informational field unless explicitly weighted. |
budget_slope |
Score trend as budget increases (positive = converges upward) |
confidence |
n / (n + k) — how much to trust the score given evidence count |
adjusted_stability_score |
stability * confidence + 0.5 * (1 - confidence) |
update_cosine_mean |
Mean of update_cosine across observations. Diagnostic only, not a stability_score component. |
dead_unit_rate / saturated_unit_rate |
Fraction of observations where dead_unit_count/saturated_unit_count is > 0.0 — how often this sample induces new dead/saturated units. Diagnostic only; feeds block-score's pathological classification, not stability_score. |
trajectory_effect_mean / trajectory_effect_sign_agreement |
Mean and sign-agreement of trajectory_effect. Diagnostic only; feeds block-score's growth/shrink classification, not stability_score. |
Components field
Each sub-score is also exposed in components so you can see why a sample was scored as it was:
Use --weight-* flags to tune individual dimensions, or use --profile (see Profiles).
Profiles
Apply a use-case preset with --profile instead of tuning weights manually. Explicit
--weight-* and --decision-score flags always override the preset.
| Profile | Weight changes | Default decision-score |
|---|---|---|
llm-judge |
evaluator ×2, model ×2 | lcb |
simulation |
budget ×2, seed ×2 | adjusted |
game-ai |
budget ×2, seed ×2, score_sign ×2; min-observations 4, min-budgets 2, min-seeds 2, min-evaluators 2 | lcb |
game-ai-single-engine |
same as game-ai, but no min-evaluators floor — for one engine evaluated at multiple depths |
lcb |
benchmark |
label ×2, evaluator ×1.5 | raw |
# LLM judge preset (equivalent to --weight-evaluators 2 --weight-models 2 --decision-score lcb)
# Override one weight from the preset
Confidence and adjusted score
confidence = n / (n + k) where k defaults to 3.0.
| n_observations | confidence (k=3) |
|---|---|
| 1 | 0.25 |
| 2 | 0.40 |
| 5 | 0.63 |
| 10 | 0.77 |
| 20 | 0.87 |
adjusted_stability_score = stability_score * confidence + 0.5 * (1 - confidence)
A sample with stability_score = 0.95 but only 2 observations gets adjusted_stability_score ≈ 0.68 — unlikely to reach the keep threshold (0.85) without more evidence.
Use --use-adjusted-score to make decisions based on the adjusted score.
Use --confidence-k to tune the convergence speed.
Minimum requirements for Keep
High stability does not guarantee sufficient evidence. Use --min-*-keep to demote underevidenced samples to Review:
score warns on stderr if a --min-evaluators-keep floor (explicit, or via a profile like
game-ai) can never be satisfied by the input — e.g. requiring 2 distinct evaluator_id
values on data that only ever has one. If you're evaluating with one engine at multiple search
depths rather than multiple engines, use --profile game-ai-single-engine instead, which sets
no min-evaluators floor.
Decisions
By default, decisions use stability_score. Five decision modes are available:
| Flag | Alias | Score used | Behaviour |
|---|---|---|---|
--decision-score raw (default) |
— | stability_score |
Raw stability. Fast; can overfit small-n. |
--decision-score adjusted |
--use-adjusted-score |
adjusted_stability_score |
Penalises low-evidence samples proportionally. |
--decision-score lcb |
--use-lcb-score |
label_agreement_lcb (label) |
Wilson LCB — most conservative. A 2/2 label match gives LCB ≈ 0.34 at 95% confidence, so it will not be kept without more evidence. |
--decision-score gold-weighted |
— | stability_score (label component reliability-weighted) |
Replaces the raw majority label vote with a gold_label-reliability-weighted vote. Falls back to identical behaviour as raw when no gold_label is present anywhere in the input. |
--decision-score latent-truth |
— | stability_score (label component EM-weighted) |
Replaces the raw majority label vote with a Dawid-Skene EM-estimated vote — infers per-evaluator reliability from disagreement patterns alone, no gold_label required. Falls back to identical behaviour as raw when the input has fewer than 2 distinct evaluators or labels. Adds latent_truth_label, latent_truth_confidence, latent_truth_label_distribution, majority_latent_conflict, evaluator_effective_n, correlated_evaluator_warning, latent_truth_converged, latent_truth_iterations, latent_truth_convergence_delta, latent_truth_demotion_reason to the output. latent_truth_confidence is EM's fitted posterior, which saturates toward 0/1 with only a few evaluators and can be confidently wrong when evaluators are correlated in their bias — it is not a correctness guarantee. evaluator_effective_n/correlated_evaluator_warning diagnose that same risk (Kish design effect on pairwise-correlated evaluator errors) but can only detect it when other evaluators reveal a correlated block's shared deviation — if the block captures the EM consensus outright, no warning fires; it's a self-consistency diagnostic, not an independent correctness check. latent_truth_converged is false when the EM loop exhausted its iteration cap without settling (latent_truth_iterations hits the cap, latent_truth_convergence_delta stays above the convergence epsilon) — the posterior is still valid but less certain; the same value applies to every sample scored in that batch, since EM runs once per batch. |
MinRequirements are always applied after the threshold comparison. For
--decision-score latent-truth, three additional opt-in, default-off safety demotions apply
after that: --demote-on-correlated-warning (demotes when correlated_evaluator_warning
fires), --min-evaluator-effective-n <f64> (demotes when evaluator_effective_n falls below
this absolute value, independent of the relative 70%-of-nominal warning above), and
--demote-on-non-convergence (demotes when latent_truth_converged is false). Each only ever
demotes Keep → Review — never further to Drop, since these are uncertainty signals, not
certainty-of-wrongness signals — and never touches stability_score itself. When a demotion
fires, latent_truth_demotion_reason names which condition (correlated_evaluator_warning,
evaluator_effective_n_below_minimum, or latent_truth_not_converged, in that priority order
if more than one fires at once). Available on score and stable-wrong-risk; no-op on
calibrate/policy/compare --policy-after, which compare raw scores directly rather than
reading decision.
| Condition | Decision |
|---|---|
| score >= 0.85 | keep |
| score <= 0.40 | drop |
| otherwise | review |
This maps onto selective classification / reject option / risk-coverage tradeoffs: rather
than forcing every sample into a class, uncertain ones are routed to review to control the risk
of what's actually accepted. keep = accept, review = abstain/defer, drop = reject for
training use. calibrate/policy operationalize this — see
calibrate — but they pick a threshold empirically against one gold-labeled
sample, not a conformal-style formal guarantee; use --heldout for a more honest (still
empirical) precision estimate. Full discussion: docs/metrics.md.
The defaults — 0.85 keep, 0.40 drop — were chosen to leave a deliberate review band.
0.85 requires strong agreement across most observations before a sample is trusted;
0.40 only rejects samples with clear, consistent disagreement. Everything between
is uncertain enough to warrant human review rather than an automatic decision.
For high-stakes training data, raise --keep-threshold to 0.90–0.95. For noisy
synthetic data where volume matters more than purity, lower it to 0.75–0.80.
Configurable via --keep-threshold and --drop-threshold. Use --confidence-level to tune the
Wilson LCB confidence level (default 0.95).
The --use-adjusted-score and --use-lcb-score boolean flags are aliases for
--decision-score adjusted and --decision-score lcb respectively, kept for backwards
compatibility. When both --decision-score and a boolean alias are specified,
--decision-score takes precedence.
explain command
Print a detailed breakdown for one sample:
sample_id: a
decision: keep
n_observations: 3
stability_score: 0.9700
confidence: 0.5000
adjusted_score: 0.7350
label_agreement_lcb:0.4380
label_margin: 1.0000
label_entropy: 0.0000
score stats:
mean: 0.8950
std: 0.0150
mad: 0.0150
iqr: 0.0300
components:
label 1.0000 ████████████████████
score_consistency 0.9850 ███████████████████
budget_robustness 0.8800 █████████████████
seed_robustness 0.9200 ██████████████████
Add --json to get the full StabilityReport as JSON.
Note: this example uses the default raw-score decision mode (
stability_score = 0.97 → keep). With--use-adjusted-score(confidence ≈ 0.50 at n=3),adjusted_score = 0.74falls below the keep threshold — the decision would be review unless--keep-thresholdis lowered. With--use-lcb-score,label_agreement_lcb ≈ 0.44also falls below 0.85 — review.
compare command
Compare two scored JSONL files by sample_id:
matched samples: 10000
mean stability: 0.7412 → 0.7801
rank stability: 0.8912 (Spearman's rho of stability_score ranks)
decision transitions (before → after):
→keep →review →drop
keep↓ 7210 311 42
review↓ 508 2101 301
drop↓ 19 104 404
top 5 regressions:
sample_001 0.9100 → 0.4400 (Δ-0.4700)
sample_382 0.8800 → 0.3900 (Δ-0.4900)
rank stability is Spearman's rank correlation (average-rank tie handling) between the
before/after stability_score orderings among matched samples — 1.0 means the relative
ranking didn't change at all even if absolute scores drifted, -1.0 means the ranking fully
reversed. Omitted (from both text and --json output) when fewer than 2 matched samples or
either side has zero variance (undefined). --json adds it as rank_stability.
Add --json for machine-readable output.
Add --components to show per-dimension mean deltas:
matched samples: 10000
mean stability: 0.7412 → 0.7801
decision transitions (before → after):
...
component deltas (mean before → after):
label 0.88 → 0.90 (+0.02)
score_consistency 0.79 → 0.86 (+0.07)
budget_robustness 0.91 → 0.72 (-0.19) ← regression
seed_robustness 0.88 → 0.89 (+0.01)
--json adds a component_deltas object with signed delta values.
summary command
samples: 1000
keep: 621 (62.1%)
review: 291 (29.1%)
drop: 88 (8.8%)
lcb_keep_demotions: 139 (stability_score >= 0.85, label_agreement_lcb < 0.85)
stability_score:
mean: 0.7412
median: 0.7810
p10 / p90: 0.4200 / 0.9600
score dispersion (mean across samples):
mad: 0.0421
iqr: 0.0812
top instability drivers (review + drop samples):
label disagreement 38%
score variance 24%
seed sensitivity 21%
budget sensitivity 17%
lcb_keep_demotions counts samples where stability_score >= keep_threshold (raw mode would
keep them) but label_agreement_lcb < keep_threshold (LCB mode would not) — the number of
samples that switching to --decision-score lcb would demote from keep. Samples already
below the threshold in raw mode are excluded. Pass --keep-threshold to match the value used
during scoring.
Use --json for CI integration:
|
filter command
In addition to --min-stability, --max-disagreement, and --decision, filter supports
diagnostic field filters:
| Flag | Keeps records where |
|---|---|
--min-label-lcb <f> |
label_agreement_lcb >= f (drop low-evidence keeps) |
--min-confidence <f> |
confidence >= f (drop low-observation-count samples) |
--max-score-mad <f> |
score_mad <= f (drop high-dispersion samples) |
--max-score-iqr <f> |
score_iqr <= f (drop high-spread samples) |
# Keep only samples with high evidence and low score dispersion
Records that lack the filtered field (e.g. label_agreement_lcb is absent when no labels were
provided) are excluded by --min-* filters and included by --max-* filters.
reliability command (experimental)
Stability measures agreement, not correctness. Use
stable-wrong-riskto quantify how many of your kept samples are consistently wrong — the most dangerous failure mode in stability-filtered datasets.
Estimate per-evaluator reliability from observation JSONL:
Reliability is the fraction of evaluations where the evaluator's label matches the reference label.
By default, the reference is the majority label across evaluators. If gold_label is set on any
observation for a sample, it is used as the reference instead — enabling ground-truth-based
reliability without changing the scoring output.
The trailing line reports two dataset-level agreement statistics:
| Field | Meaning |
|---|---|
fleiss_kappa |
Inter-rater agreement corrected for chance (nominal labels, variable raters per subject). 0 = chance, 1 = perfect, negative = worse than chance. |
krippendorff_alpha |
Agreement coefficient using the coincidence-matrix formulation for nominal labels. More general than kappa; same scale. |
Both are omitted when fewer than 2 subjects have at least 2 ratings each (undefined).
Use jq 'select(.fleiss_kappa)' to extract the summary line.
When gold_label is present, each evaluator line also includes a confusion matrix
(predicted → gold → count):
audit command
Deep diagnostic report for a scored JSONL file:
=== quietset audit ===
total: 1000
keep: 621 (62.1%)
review: 291 (29.1%)
drop: 88 (8.8%)
lcb_keep_demotions: 139 (stability >= 0.85, lcb < 0.85)
stability_score:
mean: 0.7412
median: 0.7810
p10 / p90: 0.4200 / 0.9600
top instability drivers:
label disagreement 38%
score variance 24%
sample_042 0.8201 review
sample_187 0.8490 keep
sample_003 stability=0.9100 lcb=0.3423
sample_091 budget_sensitivity=0.8200
--json output includes borderline, high_raw_low_lcb, high_score_mad, budget_sensitive,
and seed_sensitive as arrays of {sample_id, ...} objects, suitable for piping to downstream tools.
calibrate command
Find a keep_threshold that meets a precision or coverage target, using gold_label observations:
calibrate grid-searches keep_threshold from 0.99 down to 0.50 (step 0.01) and returns the
loosest threshold that meets the target. Requires gold_label on at least one observation per
sample. Returns an error if no threshold meets the target (try a lower --target-precision).
precision_ci_low/precision_ci_high is a Wilson score interval around achieved_precision at
--confidence-level (the same knob used for the lcb decision score) — it widens automatically
when few gold labels back the estimate, so don't treat achieved_precision alone as exact when
n_keep is small. The interval covers sampling noise at the chosen threshold only, not the fact
that the threshold itself was picked to hit the target on these same gold labels — it is still
optimistic. Treat it as a lower bar, not the full uncertainty; a held-out gold set is the more
conservative check. note makes this caveat machine-readable: it's always present when
--heldout was not used (as above), and present when --heldout was used but the training-set
precision at the selected threshold exceeded the held-out precision by more than 0.05 (a likely
overfit); null when --heldout was used and the gap was small enough to be ordinary sampling
noise.
Use --heldout <path> to get that more conservative check: keep_threshold is still selected
by grid search on the primary input, but achieved_precision/precision_ci_low/
precision_ci_high/coverage/n_keep/n_total are instead measured on the held-out file's own
gold_labels at that fixed threshold. For raw/adjusted/lcb/latent-truth this fully
removes the circularity — the threshold-selection circularity is gone, and latent-truth's EM
never consumes gold labels at all, so grading its inferred labels against held-out gold is a
genuine independent check even on the same samples. For gold-weighted, one layer of
circularity remains: the per-evaluator reliability weights are fit from the held-out file's own
gold_labels and then graded against those same labels, so treat its held-out precision as
better than the in-sample number but not fully independent. Not stdin-capable (-); the primary
input already claims stdin. Requires gold_label on the held-out file, or the command errors.
The reported precision can legitimately land below --target-precision when the
primary-input-selected threshold overfits — that's the correct, honest signal, not a bug.
validated_on_heldout in the output discloses whether --heldout was used, so a saved result
still says what it measured. train_precision exposes the in-sample precision at the same
threshold alongside achieved_precision (which becomes the held-out precision once --heldout
is used) — null unless --heldout was given, since otherwise it would just repeat
achieved_precision:
Use --output-format csv for a single header row + one data row instead of a JSON object:
decision_score,keep_threshold,drop_threshold,achieved_precision,precision_ci_low,precision_ci_high,coverage,n_keep,n_total,train_precision,validated_on_heldout,note,group_field,leaked_group_count,group_coverage
lcb,0.910000,0.400000,0.982000,0.943000,0.997000,0.610000,610,1000,,false,"achieved_precision and its confidence interval were selected and measured on the same gold-labeled data; the threshold search makes this optimistic. Pass --heldout <path> for an independent estimate.",,,
The original 12 columns keep their order and meaning unchanged; group_field/
leaked_group_count/group_coverage are appended at the end and are empty unless --group-by
was used with --heldout. group_coverage is named to avoid colliding with the unrelated
coverage column (precision/recall coverage, not group-field coverage) already at position 7.
Use --fail-below-target to make calibrate a CI gate: it exits non-zero (after still printing
the result) if achieved_precision falls short of --target-precision. Without --heldout
this is effectively a no-op — the grid search already guarantees achieved_precision >= --target-precision on the primary input, or fails outright with an error. With --heldout,
achieved_precision is the held-out measurement, which can legitimately fall short — this flag
turns that honest signal into something a CI pipeline can act on:
calibrate --group-by
--heldout only works if the two files are genuinely independent. If nearby samples from the
same game record (or the same opening family, conversation, etc.) end up split across the two
files, the held-out precision is measuring partly-seen data and reads more optimistic than a
genuinely independent estimate. --group-by reports that leakage — it does not build or repair
the split for you; you still prepare --heldout yourself:
warning: 1 group key(s) (source_root_id) appear in both the calibration input and --heldout; achieved_precision is optimistic
leaked source_root_id: game_00042
--group-by accepts source-root-id or opening-family (reading the source_root_id/
opening_family observation fields — see
Split-integrity fields). group_leakage is null when
nothing was checked (no --group-by, or no --heldout) — distinct from leaked_group_count: 0,
which means the check ran and found a clean split among the rows it could check. Without
--heldout, --group-by has no split to check and only prints a warning.
Two counts guard against leaked_group_count: 0 being misread as "verified clean":
- No row on either side carries the field at all (
rows_with_group: 0) — quietset warns that leakage could not be checked, rather than silently reporting a clean split. A field-name mismatch between your producer and--group-bywould otherwise read as "0 shared groups" by coincidence, not by verification. - Some, but not all, rows carry the field (
rows_missing_group > 0,coverage < 1.0) — quietset warnsincomplete_group_coverageon stderr: a leak found among the covered rows is still reported (the check never hides a leak by excluding the missing rows), but a clean result on partial coverage means only the covered portion was verified, not the whole file.
leaked_group_count is always the exact total, computed over every leaked group regardless of
dataset size. leaked_keys_sample lists at most 100 of them (sorted, so it's deterministic which
100), so a dataset with a very large number of leaked groups can't make an ordinary calibration
run's output grow without bound; leaked_keys_truncated is true when the sample is a prefix
rather than the complete set (leaked_group_count > leaked_keys_sample.len()). If you need the
complete, uncapped list, call the Rust API's leaked_group_keys(calibration, heldout, key)
directly — it's the same computation group_leakage uses internally, just without the cap.
compute_calibration's parameter list is unchanged and never grows — --group-by is implemented
via a separate compute_calibration_with_options(..., &CalibrationOptions { group_by }) entry
point in the Rust API, so existing library callers of compute_calibration are unaffected by
this feature.
Note: calibrate cannot separate stable-correct from stable-wrong samples — if a sample consistently gets the wrong label, its
stability_scoreis indistinguishable from a correct sample. Usegold_label-basedreliabilitydiagnostics to identify systematically wrong evaluators.
select command
Extract samples by diagnostic class, outputting the original scored JSONL lines (pass-through, pipeable to other commands):
|
| Class | Selects |
|---|---|
borderline |
keep_threshold ± 0.10 stability band (uncertainty zone) |
high-disagreement |
sorted by disagreement_score descending |
budget-sensitive |
sorted by budget_sensitivity descending |
seed-sensitive |
sorted by seed_sensitivity descending |
high-raw-low-lcb |
stability_score >= keep_threshold but label_agreement_lcb < keep_threshold |
high-score-mad |
sorted by score_mad descending |
Use --top N to limit output. Use --keep-threshold to adjust the band for borderline and
high-raw-low-lcb (default 0.85).
recommend command
Emit a re-evaluation suggestion for each sample that has a detectable issue, in priority order:
| Reason | Action |
|---|---|
high_raw_low_lcb |
LCB below threshold despite high raw stability → add_observations |
low_evaluator_agreement |
evaluator_agreement < 0.7 → add_evaluators |
high_seed_sensitivity |
seed_sensitivity > 0.3 → add_seeds |
high_budget_sensitivity |
budget_sensitivity > 0.3 → increase_budget |
low_model_agreement |
model_agreement < 0.7 → add_models |
low_score_consistency |
score_consistency < 0.7 → reduce_score_variance |
Each sample emits at most one recommendation (highest priority rule wins).
stable-wrong-risk command
Scores observation JSONL internally and reports kept samples whose majority_label differs from
gold_label:
Requires gold_label on observations. Sorted by stability_score descending — the most
confidently-kept wrong samples appear first. Use --top N to limit the sample list.
Use --breakdown to also report stable_wrong_rate per evaluator_id/model_id/budget
value among Keep-decision samples — spotting a specific evaluator/model/budget that shows up
disproportionately often in stably-wrong-but-kept samples:
Each breakdown is sample-level, like the top-level rate: by_evaluator[i].n_keep counts Keep
samples that touch that specific evaluator_id, not observations. A sample can touch multiple
evaluators/models/budgets at once, so rows within a breakdown do not sum to the top-level
n_keep — that's intentional, not a partition; it answers "which evaluators/models/budgets show
up in wrong-but-confident samples," not "whose fault is it." Off by default, since it clones the
input and does an extra grouping pass.
compare --policy-after
After the standard comparison output, show how decisions in the after file would change under a hypothetical decision-score policy:
policy comparison: current → lcb (keep_threshold=0.85):
→keep →review →drop
keep↓ 0 850 0
review↓ 0 2291 300
drop↓ 0 0 200
demoted by policy: 850 promoted: 0
Note:
--policy-after lcbuseslabel_agreement_lcbas a proxy for the LCB policy score. Other components are not recomputed, so results are approximate. Use for directional signal ("how many keeps would be demoted"), not precise prediction.
policy command
Sweeps keep_threshold from 0.99 down to 0.50 and reports the precision/coverage/stable-wrong-rate
trade-off at each step, so you can pick a threshold before running score:
threshold n_keep coverage
0.99 1 0.500
0.98 1 0.500
0.97 1 0.500
With gold_label present on observations, the table also gains precision and
stable_wrong_rate columns. --target-precision/--target-coverage mark the loosest
threshold meeting that target with ←. --output-format json emits one JSONL object per
threshold row instead of the formatted table (--json still works as an alias; passing both
warns and --output-format wins). --output-format csv emits one row per swept threshold with
fixed columns threshold,n_keep,coverage,precision,stable_wrong_rate,best — precision/
stable_wrong_rate are empty when no gold_label is present, and best is true only on the
row matching --target-precision/--target-coverage:
threshold,n_keep,coverage,precision,stable_wrong_rate,best
0.99,1,0.500000,1.000000,0.000000,
0.98,1,0.500000,1.000000,0.000000,true
active-review command
Ranks scored JSONL samples by re-evaluation urgency — a weighted combination of low
label_agreement_lcb, high label_entropy, high score_mad, and high budget/seed sensitivity:
|
--unstable-only skips samples already decided keep with no instability signals. Per-signal
--weight-* flags (--weight-lcb, --weight-entropy, --weight-score-mad,
--weight-budget-sensitivity, --weight-seed-sensitivity, --weight-order-sensitivity,
--weight-teacher-conflict, --weight-gradient-instability) let you emphasize the signal most
relevant to your review budget. suggested_action is one of request_gold_label,
add_evaluator, add_model, increase_budget, add_seed, add_shuffle_seed,
flag_teacher_conflict, add_init_seed (the first five renamed from the earlier
add_observations/diversify_evaluators/reduce_score_variance/add_budget/add_seeds — a
breaking change, see CHANGELOG; the last three added for trajectory stability — see
Trajectory stability fields). The 3 trajectory signals
only fire when the corresponding Observation fields (shuffle_seed, teacher_residual,
gradient_sign) were present when score ran:
| Signal | primary_reason |
suggested_action |
Cost flag |
|---|---|---|---|
Order-dependency (shuffle_seed_sensitivity) |
high_order_sensitivity |
add_shuffle_seed |
--cost-shuffle-seed |
Teacher conflict (1 - teacher_residual_stability) |
high_teacher_conflict |
flag_teacher_conflict |
--cost-gold-label |
Gradient instability (1 - gradient_sign_agreement) |
high_gradient_instability |
add_init_seed |
--cost-seed |
Five additional fields turn the urgency heuristic into an expected-value ranking:
label_agreement_flip_ratio— a distance-to-threshold ratio in[0.0, 1.0]:1.0exactly at--keep-threshold/--drop-threshold, shrinking toward0.0further away, relative tolabel_agreement's own Wilson-CI margin of error. Not a probability — onlylabel_agreement's sampling uncertainty is modeled;stability_score's other components (score dispersion, budget/seed sensitivity, model/evaluator agreement) have no uncertainty model in this codebase, so this ratio is a loose proxy whenever those other components actually drivestability_score.Nonewhen the sample has no labels.expected_coverage_gain—1 / n_totalif the sample'sdecision(read verbatim from the input, not re-derived) is notkeep, else0.0: the exact coverage change if this sample's decision flipped tokeep.expected_risk_reduction—1 / n_keepif the sample is akeepwhoselabel_agreement_lcbfalls below--keep-threshold(the same "at-risk keep" conditionstable-wrong-risk'slcb_keep_demotionscounts), else0.0.cost— the cost ofsuggested_action, from--cost-seed/--cost-budget/--cost-evaluator/--cost-gold-label(all default1.0;--cost-evaluatoralso backsadd_model).utility—label_agreement_flip_ratio * (expected_coverage_gain + expected_risk_reduction) / cost. A ranking score, not a literal expected value — ordering by it is sound (closer to threshold, higher payoff, cheaper actions rank higher), but don't read the absolute number as a calibrated expectation.
--keep-threshold/--drop-threshold (defaults 0.85/0.40, matching score's own defaults)
feed only label_agreement_flip_ratio and the at-risk-keep gate — they do not re-derive
decision. --rank-by urgency (default) sorts by urgency_score; --rank-by utility sorts by
utility instead, surfacing samples where additional review effort pays off most rather than
samples that merely look unstable.
Use --plan <path> to write a budget-constrained evaluation plan — a JSONL file, always sorted
by utility descending regardless of --rank-by, in addition to the normal stdout output.
--budget <f64> caps the total cost of entries it includes: entries are taken greedily by
utility, skipping (not stopping at) any that don't fit so a cheaper lower-utility entry later in
the list can still be included. Without --budget, --plan writes every entry (still
utility-sorted):
This is a standard greedy value-density approximation for a knapsack-style budget allocation problem — not guaranteed globally optimal, but sound for "what should the next batch of review effort go toward."
Pass --observations <path> (the same raw observation JSONL score was run on) to also size a
concrete target for suggested_action — StabilityReport only carries derived stats
(sensitivity/robustness scores), not the raw budget/seed/evaluator/model values needed to size
one, the same gap audit --observations fills for agreement stats:
Only the field matching suggested_action is populated, from that sample's own observations:
increase_budget→target_budget= the sample's max observedbudget× 2add_seed/add_init_seed→target_seed= the sample's max observedseed+ 1 (both actions size the sameseedfield — "run one more seed")add_shuffle_seed→target_shuffle_seed= the sample's max observedshuffle_seed+ 1add_evaluator→target_evaluator_slot= the sample's distinctevaluator_idcount + 1add_model→target_model_slot= the sample's distinctmodel_idcount + 1request_gold_label/flag_teacher_conflict→ no target field (nothing to size for a review/label request)
All fields are omitted without --observations, and omitted per-sample whenever the
relevant raw value is missing from the observations file — never a fabricated default.
block-score command
Groups observations by block_id (e.g. a 32-position training block) instead of sample_id
and classifies each block's trajectory stability:
block_stability = seed_effect_consistency × shuffle_direction_consistency × checkpoint_reproducibility — the product of whichever of those three factors are computable
(each requires ≥2 distinct groups on its axis: seed, shuffle_seed, model_id
respectively). classification is one of:
| Class | Meaning |
|---|---|
stable_growth |
block_stability >= --stable-threshold and net trajectory_effect_mean > 0 |
stable_shrink |
block_stability >= --stable-threshold and net trajectory_effect_mean < 0 |
seed_sensitive |
Unstable, and seed_effect_consistency is the lowest sub-threshold factor |
order_sensitive |
Unstable, and shuffle_direction_consistency is the lowest sub-threshold factor |
trajectory_sensitive |
Unstable, and checkpoint_reproducibility is the lowest sub-threshold factor |
pathological |
dead_unit_rate/saturated_unit_rate exceeds --pathological-dead-rate — overrides every other signal |
insufficient |
Not enough condition-axis diversity (or no trajectory_effect to read a growth/shrink sign from), or the block mixes more than one non-null loss_recipe — see below |
stable_growth/stable_shrink are not "good"/"bad" labels — they describe reproducibility
of trajectory_effect's sign, not whether that effect is desirable. Growth is not always an
improvement; shrink is not always a regression (a shrink with rising dead/saturated-unit rates
is caught separately by pathological, but a clean shrink could be healthy regularization).
Recommended handling, pending your own held-out/downstream validation:
| Class | Suggested handling |
|---|---|
pathological |
Exclude from training |
insufficient |
Collect more observations before deciding |
seed_sensitive |
Add another seed and re-classify |
order_sensitive |
Add another shuffle seed and re-classify |
trajectory_sensitive |
Re-evaluate at another checkpoint |
stable_growth |
Send to downstream validation — do not auto-keep |
stable_shrink |
Send to downstream validation — do not auto-drop |
--stable-threshold (default 0.85), --sensitivity-threshold (default 0.5), and
--pathological-dead-rate (default 0.5) tune the classification, same "operational default,
not a derived constant" status as score's --keep-threshold/--drop-threshold — not a
calibrated probability of anything (in particular, not of a downstream playing-strength
improvement); validate against your own held-out data and match results before trusting it to
gate training data unattended. See docs/metrics.md for why the three
consistency factors are multiplied rather than averaged. Warns on stderr (doesn't error) if no
observation carries block_id.
A block mixing more than one non-null loss_recipe is never aggregated as ordinary
variation. Different recipes confound every other axis this classification measures — a
block can't be called "seed-stable" or "pathological" if the loss function itself also changed
underneath it. block-score warns on stderr and forces classification to insufficient for
any such block (this overrides even pathological, since the dead/saturated-unit rate itself
is confounded too). A block where loss_recipe is present but missing on some observations
(all present values agreeing) only gets an "incomplete coverage" warning — classification is
unaffected. layer_id is unrelated to this: a block naturally spans multiple layers, and
mixing layer_id values within one block is normal, not a data problem.
trajectory-audit command
Diffs block-level trajectory stability between a before/after observation file (e.g. two training checkpoints) — the destructive-sample detector:
B5 stable_shrink → pathological
dead_unit_delta[ft]: +1.0000
trajectory_effect_delta: -0.4500
Matches blocks by block_id (blocks present on only one side are skipped) and reports, per
matched block: dead_unit_delta/saturated_unit_delta (per layer_id, after_rate - before_rate; a layer seen on only one side is treated as 0.0 on the other, so a brand-new
problem layer still shows its full rate as the delta), trajectory_effect_delta,
before_classification/after_classification, and reproducibility_3seed
(seed_effect_consistency on the after side, only when ≥3 distinct seed values are present
there). Sorted by the largest dead/saturated-unit delta magnitude descending — most-destructive
blocks first. --json for machine-readable JSONL output (one object per block).
--group-size N warns on stderr for any block whose distinct sample_id count doesn't match
N on either side — a malformed block grouping (e.g. a truncated 32-position block), not a
destructive-training signal. Same --stable-threshold/--sensitivity-threshold/
--pathological-dead-rate flags as block-score, including the same loss_recipe
mixed/incomplete-coverage warnings on both --before and --after.
If a block's loss_recipe differs between --before and --after (e.g. before is
"baseline", after is "teacher_conflict_masking"), the entry is not treated as an ordinary
trajectory diff — recipe change and training progress can't be told apart from the deltas
alone. Its comparable field is false, comparison_issue is "loss_recipe_mismatch", and
dead_unit_delta/saturated_unit_delta/trajectory_effect_delta/reproducibility_3seed are
left empty/absent rather than computed (before_classification/after_classification are
still each side's own report and remain meaningful). Text output prints not comparable: loss_recipe_mismatch for these blocks instead of the usual deltas.
preflight command
Checks an observation dataset's instrumentation before running the real analysis, so a
missing block_id/shuffle_seed is caught while the experiment can still be re-instrumented,
not discovered after the fact:
=== quietset preflight (for trajectory-audit) ===
input: 30 observation(s), 5 block(s)
field coverage:
block_id 30/30 (100%)
seed 24/30 (80%)
shuffle_seed 18/30 (60%)
...
ready: true (0 blocking issue(s), 2 warning(s))
warnings:
- input: shuffle_seed coverage is 18/30 (60%)
- input: block b3 has only 2 distinct seed value(s) (need >= 3 for reproducibility_3seed)
--for selects which downstream command's instrumentation requirements to check — currently
only trajectory-audit (checking block_id, seed, shuffle_seed, model_id, layer_id,
loss_recipe, trajectory_effect, update_cosine, dead_unit_count, saturated_unit_count
coverage; block_id uniqueness via run_id; and distinct seed counts per block). Pass
--after <path> to also check a later checkpoint's file — this unlocks three more checks: which
blocks exist on only one side (blocks trajectory-audit would silently skip today, with no
warning), which matched blocks have a loss_recipe mismatch and would be reported as not
comparable, and which matched blocks have a contradictory run_id identity between the two
sides (the same block_id string, but the two sides' run_ids share nothing in common — an
accidental collision between two unrelated training runs, not a real before/after pair). Without
--after, preflight runs the single-file checks only — useful precisely when checking
instrumentation before a later checkpoint even exists yet.
The block_id-uniqueness check is a new, code-enforced version of a contract that was
previously only a doc comment: block_id must be unique within one input dataset, and a
producer combining independent training runs into one file must namespace it itself (e.g.
"{run_id}:{block_id}"). preflight checks this via run_id — if a block_id group spans more
than one distinct run_id, that's reported as a likely unnamespaced collision. This check is
inert (reports nothing, not "verified unique") when run_id is absent from every observation —
preflight's output distinguishes "no issue found" from "not checkable" via
n_missing_run_id.
Blocking vs. warning, and exit codes
Every check lands in one of two buckets:
| Meaning | Exit code effect | |
|---|---|---|
Blocking (blocking_issues) |
trajectory-audit's output on this input cannot be trusted at all, not just degraded |
Non-zero by default |
Warning (warnings) |
A specific metric is degraded, but the rest of the analysis is still meaningful | Never affects exit code |
Blocking: block_id present on 0% of observations (nothing can be grouped into blocks at all), a
block_id spanning more than one run_id (a namespacing collision, so the block's own numbers
are already meaningless), a block's loss_recipe internally mixed, zero blocks corresponding
between --before/--after, a matched block's run_id identity contradicting itself across
sides, or a matched block's loss_recipe mismatched across sides.
Warning: partial coverage on any field (including block_id — only total absence blocks), a
block with fewer than 3 distinct seed values (reproducibility_3seed just doesn't compute for
it — everything else does), some (not all) blocks present on only one side, or partial
run_id/loss_recipe coverage within a block.
Contract: --before/--after are assumed to be two checkpoints of the same training run.
A given block_id is expected to carry the same run_id on both sides — trajectory-audit has
no support for comparing reproducibility across distinct runs by block_id alone. This is why
a disjoint run_id set between --before and --after is blocking rather than a warning: it
reads as an accidental block_id collision between two unrelated runs, not a legitimate
cross-run comparison. If you need to compare across runs, block_id can't express that on its
own — it would need an explicit shared key (a lineage/recipe/seed identifier) that doesn't exist
in this codebase today; treat cross-run comparison as unsupported rather than relying on
disagreeing run_ids to mean "different but comparable."
ready (true when blocking_issues is empty), blocking_issue_count, warning_count,
blocking_issues, warnings, comparable_block_count, and incomparable_block_count (the
latter two null without --after) are always in the --json output, so a CI script can check
ready directly instead of parsing prose:
|
Add --json for a single pretty-printed PreflightReport object instead of text.
Exit codes:
| Exit code | Meaning |
|---|---|
0 |
No blocking issues (warnings may still be present) |
1 |
Blocking issue(s) found (or --report-only was not passed) |
| non-zero, no report printed | Usage error: empty/unparseable input, bad flags, etc. — the existing quietset error convention |
Pass --report-only to always exit 0 and just review the report yourself, even with blocking
issues present — the report (ready, blocking_issue_count, blocking_issues, everything) is
byte-for-byte identical either way; the flag only changes what happens after it's printed.
Seed shortages and one-sided blocks alone never block, matching the block-score/
trajectory-audit convention that a coverage gap is a warning, not a failure — only findings
that make the eventual trajectory-audit run's output actively untrustworthy do.
Rust API
use ;
let obs = vec!;
let reports = score_all;
println!;
Streaming API
use ;
let mut scorer = new;
for obs in observations
if let Some = scorer.flush
Where these numbers come from
quietset is not an implementation of one paper. Most of its per-sample building blocks are
standard, named statistics; stability_score and the keep/review/drop policy built on top of
them are quietset's own design — useful, but not a theoretical guarantee. This table is a
compact classification; for the full formula, assumptions, and failure modes behind each
metric, see docs/metrics.md.
Established statistics (faithful implementations, full formulas in the sections linked):
| Field | Basis |
|---|---|
label_agreement_lcb |
Wilson score interval lower bound — see Stability score |
fleiss_kappa |
Fleiss' kappa, generalized to variable raters per subject — see reliability |
krippendorff_alpha |
Krippendorff's alpha, nominal-labels variant — see reliability |
label_entropy |
Normalized Shannon entropy over the labels observed for that sample — see Stability score |
score_std / score_mad / score_iqr |
Population standard deviation / median-based MAD / linear-interpolation IQR — see Stability score |
Model-based estimator:
| Field | Basis |
|---|---|
latent_truth_* |
Dawid-Skene-style EM — per-evaluator confusion matrix + label posterior, fit with no gold_label needed. evaluator_effective_n and correlated_evaluator_warning are quietset's own diagnostics computed on top of the converged EM output, not part of the original Dawid-Skene method — see Decisions |
Empirical diagnostics (measured directly against gold_label, not modeled):
| Field | Basis |
|---|---|
stable_wrong_rate_among_keep |
Direct empirical rate — see stable-wrong-risk |
stable-wrong-risk --breakdown |
Same rate, grouped by evaluator/model/budget — see stable-wrong-risk |
quietset-specific heuristics (compose the above into an operational decision — not citable statistical methods on their own):
| Field | Basis |
|---|---|
stability_score |
Weighted mean of the sub-scores above — see Stability score |
confidence / adjusted_stability_score |
Evidence-count shrinkage toward neutral 0.5 — see Confidence and adjusted score |
score_sign_agreement |
Domain-aware complement metric for signed, zero-centered scores — see Stability score |
keep / drop thresholds (0.85 / 0.40) |
Operational defaults, not derived constants — see Decisions |
calibrate/policy are conceptually close to selective-prediction / risk-coverage ideas
(reject uncertain samples to review rather than forcing a keep-or-drop call on everything) —
but they are not a conformal-style formal guarantee. A threshold is chosen empirically
against one gold-labeled sample, so treat achieved_precision as measured, not proven; see
the in-sample-optimism note and --heldout explanation in calibrate.
Compared to adjacent tools
| Tool | What it does | How quietset differs |
|---|---|---|
| Cleanlab | Python library that detects label errors using trained classifiers and confident learning. | quietset needs no model training and makes no task-specific assumptions. It filters by cross-run stability rather than estimated label quality. |
| Label Studio | Web-based annotation platform for labelling images, text, audio, and time series. | quietset is a CLI/library primitive, not an annotation UI. It measures stability of labels already produced by other tools. |
| pandas / polars | General-purpose data manipulation libraries. | quietset provides a purpose-built stability schema — decisions, per-dimension sub-scores, confidence, instability diagnostics — that would otherwise require substantial custom code. |
| Great Expectations / Soda | Data quality frameworks that validate data against rules (nulls, ranges, schema). | Those tools check whether data conforms to a schema. quietset checks whether labels or scores are consistent across repeated evaluations. |
| scipy.stats / sklearn metrics | Statistical functions such as Cohen's kappa and Fleiss' kappa. | quietset wraps similar ideas into a composable pipeline primitive with JSONL I/O, per-sample reports, confidence adjustment, and configurable thresholds. |
| LLM evaluation frameworks (RAGAS, DeepEval) | Frameworks that score LLM outputs against reference answers using model-based judges. | quietset is judge-agnostic. It takes whatever scores or labels your judges produce and measures agreement across runs, budgets, models, or seeds. |
Python bindings
crates/quietset-py provides Python bindings via pyo3 + maturin.
Status: alpha / experimental — the core scoring API is wrapped but the interface may change.
It is excluded from the Cargo workspace and from CI (cargo build/test --workspace never touches
it); build and test it manually from within crates/quietset-py.
&&
=
The bindings currently expose a single function, score_jsonl, which scores a JSONL string with
default settings and returns a JSONL string of results. The CLI is the stable interface with the
full set of commands and options; use Python bindings for embedding basic scoring in existing
Python pipelines where spawning a subprocess is impractical.
License
MIT OR Apache-2.0