quietset
A model-agnostic stability filter — keeps samples whose labels or scores remain consistent across evaluators, budgets, seeds, and model checkpoints.
quietset is not a model trainer, annotation platform, or image-quality auditor. It is a small stability-filtering primitive designed to compose with other tools.
Note: quietset measures stability, not correctness. A sample can score high because evaluators consistently agree on a wrong answer. Use
gold_label-based reliability or--decision-score lcbto add evidence-based conservatism.
Use cases
Game AI / search training data
Multiple engines, depths, or seeds evaluate the same position. Keep only positions where the evaluation is stable — consistent labels and scores regardless of search parameters.
LLM judge pipelines
Multiple judge models or prompts evaluate the same response. Keep only responses where judges consistently agree, using Wilson LCB to guard against low-n flukes.
Synthetic / simulation data
Scores or rewards vary across seeds, budgets, or model checkpoints. Keep samples whose quality signal is robust to these variations.
|
Installation
CLI examples
# Score observations
# Filter to stable samples
# Filter by decision
# Pipeline from stdin
|
# Aggregate statistics
# Machine-readable summary for CI
|
# Explain why a specific sample was scored the way it was
# Compare two scored files (e.g. before/after a model update)
# Per-evaluator reliability (experimental)
# CSV output
# Weight label agreement 2x, ignore score variance
# Penalise low-evidence samples: decisions use confidence-adjusted score
# Penalise low-evidence samples: Wilson LCB on label agreement (most conservative)
# Explicit --decision-score flag (preferred for scripting; --use-* are aliases)
# Apply a use-case preset (sets weight and decision-score defaults)
# Require at least 3 observations and 2 evaluators before Keep
# Filter by LCB, confidence, and dispersion
# Compare with per-component deltas (spot regressions)
# Deep diagnostic audit report
|
|
# Extract samples by diagnostic class for human review
# Get re-evaluation recommendations
# Compute risk of stably-wrong kept samples
# Compare with hypothetical policy applied to after file
# Calibrate keep_threshold from gold labels to meet a precision target
Command reference
| Command | Input | What it does |
|---|---|---|
score |
observation JSONL/CSV | Compute per-sample stability scores and decisions |
filter |
scored JSONL | Keep samples by stability, decision, LCB, confidence, or dispersion |
summary |
scored JSONL | Aggregate statistics; lcb_keep_demotions; --json for CI |
explain |
scored JSONL | Per-sample component breakdown with visual bars |
compare |
2 scored JSONL | Before/after transition matrix, regressions, component deltas, policy comparison |
reliability |
observation JSONL | Per-evaluator reliability, confusion matrix, Fleiss kappa, Krippendorff alpha |
audit |
scored JSONL | Deep diagnostic report: borderline, LCB risk, sensitivity lists |
select |
scored JSONL | Extract samples by class for human review queues (pipeable) |
recommend |
scored JSONL | Per-sample re-evaluation suggestions with reasons |
stable-wrong-risk |
observation JSONL | Rate of stably-wrong kept samples (requires gold_label) |
calibrate |
observation JSONL | Find keep_threshold meeting a precision/coverage target |
policy |
observation JSONL | Sweep keep_threshold and show the precision/coverage trade-off table |
active-review |
scored JSONL | Rank samples by re-evaluation urgency (low LCB, high entropy, dispersion, sensitivity) |
Output formats
Output conventions differ by command — deliberately, since some commands are built for human
inspection and some for pipeline composition (see select and
--embed-stats). There is no single unified format; use this table to know
what to expect from each command before scripting against it:
| Command | Default | Flag(s) | Notes |
|---|---|---|---|
score |
JSONL | --output-format jsonl|csv |
csv is a terminal/export format — see below |
filter |
JSONL (pass-through) | none | Always echoes original input lines unchanged |
select |
JSONL (pass-through) | none | Always echoes original input lines unchanged |
reliability |
JSONL | none | One object per evaluator, plus an optional trailing kappa/alpha line |
active-review |
JSONL | none | One object per ranked sample |
recommend |
JSONL | --text |
Only command where JSONL is the default and text is the opt-in |
stable-wrong-risk |
single pretty JSON object | none | |
calibrate |
single pretty JSON object | --output-format json|csv |
csv is a single header row + one data row |
summary |
text | --json (single pretty object) |
|
explain |
text | --json (single pretty object) |
|
compare |
text | --json (single pretty object) |
|
audit |
text | --json (single pretty object) |
|
policy |
text table | --output-format text|json|csv (legacy --json still works as an alias for json) |
json is JSONL, one line per swept threshold; csv has fixed columns (empty cells when no gold_label) |
CSV is a dead end for piping. score --output-format csv, calibrate --output-format csv,
and policy --output-format csv exist purely for spreadsheets/BI tools. No other quietset
command can parse CSV back in (filter/summary/explain/compare/audit/select/
recommend/active-review all expect JSONL StabilityReports; stable-wrong-risk/calibrate/
reliability/policy expect JSONL Observations). If you plan to pipe score's output into
another quietset command, use the default jsonl, not csv.
Input JSONL format
All fields except sample_id are optional. gold_label provides the known-correct label for
a sample; when present, the reliability command uses it as ground truth instead of majority vote.
Output JSONL format
Key fields in the output (optional fields are omitted when not computable):
Optional fields are omitted when not computable (e.g. label_agreement_lcb only appears when
labels are present; score_mad / score_iqr require at least two numeric scores).
Stability score
The stability_score is a value in [0.0, 1.0]:
1.0= highly stable0.0= highly unstable
It is the weighted mean of available sub-scores (all in [0.0, 1.0]):
| Component | Value |
|---|---|
label_agreement |
fraction of observations with the majority label |
score_consistency |
1 - normalized_score_std |
budget_robustness |
1 - budget_sensitivity |
seed_robustness |
1 - seed_sensitivity |
model_agreement |
label agreement across models |
evaluator_agreement |
label agreement across evaluators |
score_sign_agreement |
fraction of numeric scores sharing the majority sign — weight defaults to 0 (excluded); opt in with --weight-score-sign (--profile game-ai enables it at ×2 by default) |
Missing dimensions (e.g. no labels, no budgets) are excluded from the mean.
Single observations receive stability_score = 0.5 (review by default).
Additional diagnostic fields on StabilityReport:
| Field | Meaning |
|---|---|
label_margin |
(majority_count - runner_up_count) / total. 0.0 = perfectly split |
label_entropy |
Normalised Shannon entropy [0, 1]. 1.0 = uniform label distribution |
label_agreement_lcb |
Wilson confidence interval lower bound of label_agreement. More conservative than raw label_agreement — guards against low-n coincidences. |
score_mad |
Median absolute deviation of numeric scores. More robust to outliers than score_std. |
score_iqr |
Interquartile range (Q3 − Q1) of numeric scores. |
score_sign_agreement |
Fraction of numeric scores sharing the majority sign (negative / zero / positive). Domain-aware complement to score_std/score_mad/score_iqr for signed, zero-centered scores (chess/shogi-style eval): [+0.01, -0.01] reads as near-perfectly stable by magnitude alone but is a coin-flip on sign. Always computed but excluded from stability_score by default (--weight-score-sign 0); pass --weight-score-sign <f> to make it contribute. --profile game-ai enables it at ×2 by default — for every other profile, this stays a purely informational field unless explicitly weighted. |
budget_slope |
Score trend as budget increases (positive = converges upward) |
confidence |
n / (n + k) — how much to trust the score given evidence count |
adjusted_stability_score |
stability * confidence + 0.5 * (1 - confidence) |
Components field
Each sub-score is also exposed in components so you can see why a sample was scored as it was:
Use --weight-* flags to tune individual dimensions, or use --profile (see Profiles).
Profiles
Apply a use-case preset with --profile instead of tuning weights manually. Explicit
--weight-* and --decision-score flags always override the preset.
| Profile | Weight changes | Default decision-score |
|---|---|---|
llm-judge |
evaluator ×2, model ×2 | lcb |
simulation |
budget ×2, seed ×2 | adjusted |
game-ai |
budget ×2, seed ×2, score_sign ×2; min-observations 4, min-budgets 2, min-seeds 2, min-evaluators 2 | lcb |
benchmark |
label ×2, evaluator ×1.5 | raw |
# LLM judge preset (equivalent to --weight-evaluators 2 --weight-models 2 --decision-score lcb)
# Override one weight from the preset
Confidence and adjusted score
confidence = n / (n + k) where k defaults to 3.0.
| n_observations | confidence (k=3) |
|---|---|
| 1 | 0.25 |
| 2 | 0.40 |
| 5 | 0.63 |
| 10 | 0.77 |
| 20 | 0.87 |
adjusted_stability_score = stability_score * confidence + 0.5 * (1 - confidence)
A sample with stability_score = 0.95 but only 2 observations gets adjusted_stability_score ≈ 0.68 — unlikely to reach the keep threshold (0.85) without more evidence.
Use --use-adjusted-score to make decisions based on the adjusted score.
Use --confidence-k to tune the convergence speed.
Minimum requirements for Keep
High stability does not guarantee sufficient evidence. Use --min-*-keep to demote underevidenced samples to Review:
Decisions
By default, decisions use stability_score. Five decision modes are available:
| Flag | Alias | Score used | Behaviour |
|---|---|---|---|
--decision-score raw (default) |
— | stability_score |
Raw stability. Fast; can overfit small-n. |
--decision-score adjusted |
--use-adjusted-score |
adjusted_stability_score |
Penalises low-evidence samples proportionally. |
--decision-score lcb |
--use-lcb-score |
label_agreement_lcb (label) |
Wilson LCB — most conservative. A 2/2 label match gives LCB ≈ 0.34 at 95% confidence, so it will not be kept without more evidence. |
--decision-score gold-weighted |
— | stability_score (label component reliability-weighted) |
Replaces the raw majority label vote with a gold_label-reliability-weighted vote. Falls back to identical behaviour as raw when no gold_label is present anywhere in the input. |
--decision-score latent-truth |
— | stability_score (label component EM-weighted) |
Replaces the raw majority label vote with a Dawid-Skene EM-estimated vote — infers per-evaluator reliability from disagreement patterns alone, no gold_label required. Falls back to identical behaviour as raw when the input has fewer than 2 distinct evaluators or labels. Adds latent_truth_label, latent_truth_confidence, latent_truth_label_distribution, majority_latent_conflict, evaluator_effective_n, correlated_evaluator_warning, latent_truth_converged, latent_truth_iterations, latent_truth_convergence_delta, latent_truth_demotion_reason to the output. latent_truth_confidence is EM's fitted posterior, which saturates toward 0/1 with only a few evaluators and can be confidently wrong when evaluators are correlated in their bias — it is not a correctness guarantee. evaluator_effective_n/correlated_evaluator_warning diagnose that same risk (Kish design effect on pairwise-correlated evaluator errors) but can only detect it when other evaluators reveal a correlated block's shared deviation — if the block captures the EM consensus outright, no warning fires; it's a self-consistency diagnostic, not an independent correctness check. latent_truth_converged is false when the EM loop exhausted its iteration cap without settling (latent_truth_iterations hits the cap, latent_truth_convergence_delta stays above the convergence epsilon) — the posterior is still valid but less certain; the same value applies to every sample scored in that batch, since EM runs once per batch. |
MinRequirements are always applied after the threshold comparison. For
--decision-score latent-truth, three additional opt-in, default-off safety demotions apply
after that: --demote-on-correlated-warning (demotes when correlated_evaluator_warning
fires), --min-evaluator-effective-n <f64> (demotes when evaluator_effective_n falls below
this absolute value, independent of the relative 70%-of-nominal warning above), and
--demote-on-non-convergence (demotes when latent_truth_converged is false). Each only ever
demotes Keep → Review — never further to Drop, since these are uncertainty signals, not
certainty-of-wrongness signals — and never touches stability_score itself. When a demotion
fires, latent_truth_demotion_reason names which condition (correlated_evaluator_warning,
evaluator_effective_n_below_minimum, or latent_truth_not_converged, in that priority order
if more than one fires at once). Available on score and stable-wrong-risk; no-op on
calibrate/policy/compare --policy-after, which compare raw scores directly rather than
reading decision.
| Condition | Decision |
|---|---|
| score >= 0.85 | keep |
| score <= 0.40 | drop |
| otherwise | review |
The defaults — 0.85 keep, 0.40 drop — were chosen to leave a deliberate review band.
0.85 requires strong agreement across most observations before a sample is trusted;
0.40 only rejects samples with clear, consistent disagreement. Everything between
is uncertain enough to warrant human review rather than an automatic decision.
For high-stakes training data, raise --keep-threshold to 0.90–0.95. For noisy
synthetic data where volume matters more than purity, lower it to 0.75–0.80.
Configurable via --keep-threshold and --drop-threshold. Use --confidence-level to tune the
Wilson LCB confidence level (default 0.95).
The --use-adjusted-score and --use-lcb-score boolean flags are aliases for
--decision-score adjusted and --decision-score lcb respectively, kept for backwards
compatibility. When both --decision-score and a boolean alias are specified,
--decision-score takes precedence.
explain command
Print a detailed breakdown for one sample:
sample_id: a
decision: keep
n_observations: 3
stability_score: 0.9700
confidence: 0.5000
adjusted_score: 0.7350
label_agreement_lcb:0.4380
label_margin: 1.0000
label_entropy: 0.0000
score stats:
mean: 0.8950
std: 0.0150
mad: 0.0150
iqr: 0.0300
components:
label 1.0000 ████████████████████
score_consistency 0.9850 ███████████████████
budget_robustness 0.8800 █████████████████
seed_robustness 0.9200 ██████████████████
Add --json to get the full StabilityReport as JSON.
Note: this example uses the default raw-score decision mode (
stability_score = 0.97 → keep). With--use-adjusted-score(confidence ≈ 0.50 at n=3),adjusted_score = 0.74falls below the keep threshold — the decision would be review unless--keep-thresholdis lowered. With--use-lcb-score,label_agreement_lcb ≈ 0.44also falls below 0.85 — review.
compare command
Compare two scored JSONL files by sample_id:
matched samples: 10000
mean stability: 0.7412 → 0.7801
decision transitions (before → after):
→keep →review →drop
keep↓ 7210 311 42
review↓ 508 2101 301
drop↓ 19 104 404
top 5 regressions:
sample_001 0.9100 → 0.4400 (Δ-0.4700)
sample_382 0.8800 → 0.3900 (Δ-0.4900)
Add --json for machine-readable output.
Add --components to show per-dimension mean deltas:
matched samples: 10000
mean stability: 0.7412 → 0.7801
decision transitions (before → after):
...
component deltas (mean before → after):
label 0.88 → 0.90 (+0.02)
score_consistency 0.79 → 0.86 (+0.07)
budget_robustness 0.91 → 0.72 (-0.19) ← regression
seed_robustness 0.88 → 0.89 (+0.01)
--json adds a component_deltas object with signed delta values.
summary command
samples: 1000
keep: 621 (62.1%)
review: 291 (29.1%)
drop: 88 (8.8%)
lcb_keep_demotions: 139 (stability_score >= 0.85, label_agreement_lcb < 0.85)
stability_score:
mean: 0.7412
median: 0.7810
p10 / p90: 0.4200 / 0.9600
score dispersion (mean across samples):
mad: 0.0421
iqr: 0.0812
top instability drivers (review + drop samples):
label disagreement 38%
score variance 24%
seed sensitivity 21%
budget sensitivity 17%
lcb_keep_demotions counts samples where stability_score >= keep_threshold (raw mode would
keep them) but label_agreement_lcb < keep_threshold (LCB mode would not) — the number of
samples that switching to --decision-score lcb would demote from keep. Samples already
below the threshold in raw mode are excluded. Pass --keep-threshold to match the value used
during scoring.
Use --json for CI integration:
|
filter command
In addition to --min-stability, --max-disagreement, and --decision, filter supports
diagnostic field filters:
| Flag | Keeps records where |
|---|---|
--min-label-lcb <f> |
label_agreement_lcb >= f (drop low-evidence keeps) |
--min-confidence <f> |
confidence >= f (drop low-observation-count samples) |
--max-score-mad <f> |
score_mad <= f (drop high-dispersion samples) |
--max-score-iqr <f> |
score_iqr <= f (drop high-spread samples) |
# Keep only samples with high evidence and low score dispersion
Records that lack the filtered field (e.g. label_agreement_lcb is absent when no labels were
provided) are excluded by --min-* filters and included by --max-* filters.
reliability command (experimental)
Stability measures agreement, not correctness. Use
stable-wrong-riskto quantify how many of your kept samples are consistently wrong — the most dangerous failure mode in stability-filtered datasets.
Estimate per-evaluator reliability from observation JSONL:
Reliability is the fraction of evaluations where the evaluator's label matches the reference label.
By default, the reference is the majority label across evaluators. If gold_label is set on any
observation for a sample, it is used as the reference instead — enabling ground-truth-based
reliability without changing the scoring output.
The trailing line reports two dataset-level agreement statistics:
| Field | Meaning |
|---|---|
fleiss_kappa |
Inter-rater agreement corrected for chance (nominal labels, variable raters per subject). 0 = chance, 1 = perfect, negative = worse than chance. |
krippendorff_alpha |
Agreement coefficient using the coincidence-matrix formulation for nominal labels. More general than kappa; same scale. |
Both are omitted when fewer than 2 subjects have at least 2 ratings each (undefined).
Use jq 'select(.fleiss_kappa)' to extract the summary line.
When gold_label is present, each evaluator line also includes a confusion matrix
(predicted → gold → count):
audit command
Deep diagnostic report for a scored JSONL file:
=== quietset audit ===
total: 1000
keep: 621 (62.1%)
review: 291 (29.1%)
drop: 88 (8.8%)
lcb_keep_demotions: 139 (stability >= 0.85, lcb < 0.85)
stability_score:
mean: 0.7412
median: 0.7810
p10 / p90: 0.4200 / 0.9600
top instability drivers:
label disagreement 38%
score variance 24%
sample_042 0.8201 review
sample_187 0.8490 keep
sample_003 stability=0.9100 lcb=0.3423
sample_091 budget_sensitivity=0.8200
--json output includes borderline, high_raw_low_lcb, high_score_mad, budget_sensitive,
and seed_sensitive as arrays of {sample_id, ...} objects, suitable for piping to downstream tools.
calibrate command
Find a keep_threshold that meets a precision or coverage target, using gold_label observations:
calibrate grid-searches keep_threshold from 0.99 down to 0.50 (step 0.01) and returns the
loosest threshold that meets the target. Requires gold_label on at least one observation per
sample. Returns an error if no threshold meets the target (try a lower --target-precision).
precision_ci_low/precision_ci_high is a Wilson score interval around achieved_precision at
--confidence-level (the same knob used for the lcb decision score) — it widens automatically
when few gold labels back the estimate, so don't treat achieved_precision alone as exact when
n_keep is small. The interval covers sampling noise at the chosen threshold only, not the fact
that the threshold itself was picked to hit the target on these same gold labels — it is still
optimistic. Treat it as a lower bar, not the full uncertainty; a held-out gold set is the more
conservative check. note makes this caveat machine-readable: it's always present when
--heldout was not used (as above), and present when --heldout was used but the training-set
precision at the selected threshold exceeded the held-out precision by more than 0.05 (a likely
overfit); null when --heldout was used and the gap was small enough to be ordinary sampling
noise.
Use --heldout <path> to get that more conservative check: keep_threshold is still selected
by grid search on the primary input, but achieved_precision/precision_ci_low/
precision_ci_high/coverage/n_keep/n_total are instead measured on the held-out file's own
gold_labels at that fixed threshold. For raw/adjusted/lcb/latent-truth this fully
removes the circularity — the threshold-selection circularity is gone, and latent-truth's EM
never consumes gold labels at all, so grading its inferred labels against held-out gold is a
genuine independent check even on the same samples. For gold-weighted, one layer of
circularity remains: the per-evaluator reliability weights are fit from the held-out file's own
gold_labels and then graded against those same labels, so treat its held-out precision as
better than the in-sample number but not fully independent. Not stdin-capable (-); the primary
input already claims stdin. Requires gold_label on the held-out file, or the command errors.
The reported precision can legitimately land below --target-precision when the
primary-input-selected threshold overfits — that's the correct, honest signal, not a bug.
validated_on_heldout in the output discloses whether --heldout was used, so a saved result
still says what it measured. train_precision exposes the in-sample precision at the same
threshold alongside achieved_precision (which becomes the held-out precision once --heldout
is used) — null unless --heldout was given, since otherwise it would just repeat
achieved_precision:
Use --output-format csv for a single header row + one data row instead of a JSON object:
decision_score,keep_threshold,drop_threshold,achieved_precision,precision_ci_low,precision_ci_high,coverage,n_keep,n_total,train_precision,validated_on_heldout,note
lcb,0.910000,0.400000,0.982000,0.943000,0.997000,0.610000,610,1000,,false,"achieved_precision and its confidence interval were selected and measured on the same gold-labeled data; the threshold search makes this optimistic. Pass --heldout <path> for an independent estimate."
Use --fail-below-target to make calibrate a CI gate: it exits non-zero (after still printing
the result) if achieved_precision falls short of --target-precision. Without --heldout
this is effectively a no-op — the grid search already guarantees achieved_precision >= --target-precision on the primary input, or fails outright with an error. With --heldout,
achieved_precision is the held-out measurement, which can legitimately fall short — this flag
turns that honest signal into something a CI pipeline can act on:
Note: calibrate cannot separate stable-correct from stable-wrong samples — if a sample consistently gets the wrong label, its
stability_scoreis indistinguishable from a correct sample. Usegold_label-basedreliabilitydiagnostics to identify systematically wrong evaluators.
select command
Extract samples by diagnostic class, outputting the original scored JSONL lines (pass-through, pipeable to other commands):
|
| Class | Selects |
|---|---|
borderline |
keep_threshold ± 0.10 stability band (uncertainty zone) |
high-disagreement |
sorted by disagreement_score descending |
budget-sensitive |
sorted by budget_sensitivity descending |
seed-sensitive |
sorted by seed_sensitivity descending |
high-raw-low-lcb |
stability_score >= keep_threshold but label_agreement_lcb < keep_threshold |
high-score-mad |
sorted by score_mad descending |
Use --top N to limit output. Use --keep-threshold to adjust the band for borderline and
high-raw-low-lcb (default 0.85).
recommend command
Emit a re-evaluation suggestion for each sample that has a detectable issue, in priority order:
| Reason | Action |
|---|---|
high_raw_low_lcb |
LCB below threshold despite high raw stability → add_observations |
low_evaluator_agreement |
evaluator_agreement < 0.7 → add_evaluators |
high_seed_sensitivity |
seed_sensitivity > 0.3 → add_seeds |
high_budget_sensitivity |
budget_sensitivity > 0.3 → increase_budget |
low_model_agreement |
model_agreement < 0.7 → add_models |
low_score_consistency |
score_consistency < 0.7 → reduce_score_variance |
Each sample emits at most one recommendation (highest priority rule wins).
stable-wrong-risk command
Scores observation JSONL internally and reports kept samples whose majority_label differs from
gold_label:
Requires gold_label on observations. Sorted by stability_score descending — the most
confidently-kept wrong samples appear first. Use --top N to limit the sample list.
Use --breakdown to also report stable_wrong_rate per evaluator_id/model_id/budget
value among Keep-decision samples — spotting a specific evaluator/model/budget that shows up
disproportionately often in stably-wrong-but-kept samples:
Each breakdown is sample-level, like the top-level rate: by_evaluator[i].n_keep counts Keep
samples that touch that specific evaluator_id, not observations. A sample can touch multiple
evaluators/models/budgets at once, so rows within a breakdown do not sum to the top-level
n_keep — that's intentional, not a partition; it answers "which evaluators/models/budgets show
up in wrong-but-confident samples," not "whose fault is it." Off by default, since it clones the
input and does an extra grouping pass.
compare --policy-after
After the standard comparison output, show how decisions in the after file would change under a hypothetical decision-score policy:
policy comparison: current → lcb (keep_threshold=0.85):
→keep →review →drop
keep↓ 0 850 0
review↓ 0 2291 300
drop↓ 0 0 200
demoted by policy: 850 promoted: 0
Note:
--policy-after lcbuseslabel_agreement_lcbas a proxy for the LCB policy score. Other components are not recomputed, so results are approximate. Use for directional signal ("how many keeps would be demoted"), not precise prediction.
policy command
Sweeps keep_threshold from 0.99 down to 0.50 and reports the precision/coverage/stable-wrong-rate
trade-off at each step, so you can pick a threshold before running score:
threshold n_keep coverage
0.99 1 0.500
0.98 1 0.500
0.97 1 0.500
With gold_label present on observations, the table also gains precision and
stable_wrong_rate columns. --target-precision/--target-coverage mark the loosest
threshold meeting that target with ←. --output-format json emits one JSONL object per
threshold row instead of the formatted table (--json still works as an alias; passing both
warns and --output-format wins). --output-format csv emits one row per swept threshold with
fixed columns threshold,n_keep,coverage,precision,stable_wrong_rate,best — precision/
stable_wrong_rate are empty when no gold_label is present, and best is true only on the
row matching --target-precision/--target-coverage:
threshold,n_keep,coverage,precision,stable_wrong_rate,best
0.99,1,0.500000,1.000000,0.000000,
0.98,1,0.500000,1.000000,0.000000,true
active-review command
Ranks scored JSONL samples by re-evaluation urgency — a weighted combination of low
label_agreement_lcb, high label_entropy, high score_mad, and high budget/seed sensitivity:
|
--unstable-only skips samples already decided keep with no instability signals. Per-signal
--weight-* flags (--weight-lcb, --weight-entropy, --weight-score-mad,
--weight-budget-sensitivity, --weight-seed-sensitivity) let you emphasize the signal most
relevant to your review budget. suggested_action is one of request_gold_label,
add_evaluator, add_model, increase_budget, add_seed (renamed from the earlier
add_observations/diversify_evaluators/reduce_score_variance/add_budget/add_seeds — a
breaking change, see CHANGELOG).
Five additional fields turn the urgency heuristic into an expected-value ranking:
label_agreement_flip_ratio— a distance-to-threshold ratio in[0.0, 1.0]:1.0exactly at--keep-threshold/--drop-threshold, shrinking toward0.0further away, relative tolabel_agreement's own Wilson-CI margin of error. Not a probability — onlylabel_agreement's sampling uncertainty is modeled;stability_score's other components (score dispersion, budget/seed sensitivity, model/evaluator agreement) have no uncertainty model in this codebase, so this ratio is a loose proxy whenever those other components actually drivestability_score.Nonewhen the sample has no labels.expected_coverage_gain—1 / n_totalif the sample'sdecision(read verbatim from the input, not re-derived) is notkeep, else0.0: the exact coverage change if this sample's decision flipped tokeep.expected_risk_reduction—1 / n_keepif the sample is akeepwhoselabel_agreement_lcbfalls below--keep-threshold(the same "at-risk keep" conditionstable-wrong-risk'slcb_keep_demotionscounts), else0.0.cost— the cost ofsuggested_action, from--cost-seed/--cost-budget/--cost-evaluator/--cost-gold-label(all default1.0;--cost-evaluatoralso backsadd_model).utility—label_agreement_flip_ratio * (expected_coverage_gain + expected_risk_reduction) / cost. A ranking score, not a literal expected value — ordering by it is sound (closer to threshold, higher payoff, cheaper actions rank higher), but don't read the absolute number as a calibrated expectation.
--keep-threshold/--drop-threshold (defaults 0.85/0.40, matching score's own defaults)
feed only label_agreement_flip_ratio and the at-risk-keep gate — they do not re-derive
decision. --rank-by urgency (default) sorts by urgency_score; --rank-by utility sorts by
utility instead, surfacing samples where additional review effort pays off most rather than
samples that merely look unstable.
Use --plan <path> to write a budget-constrained evaluation plan — a JSONL file, always sorted
by utility descending regardless of --rank-by, in addition to the normal stdout output.
--budget <f64> caps the total cost of entries it includes: entries are taken greedily by
utility, skipping (not stopping at) any that don't fit so a cheaper lower-utility entry later in
the list can still be included. Without --budget, --plan writes every entry (still
utility-sorted):
This is a standard greedy value-density approximation for a knapsack-style budget allocation problem — not guaranteed globally optimal, but sound for "what should the next batch of review effort go toward."
Rust API
use ;
let obs = vec!;
let reports = score_all;
println!;
Streaming API
use ;
let mut scorer = new;
for obs in observations
if let Some = scorer.flush
Compared to adjacent tools
| Tool | What it does | How quietset differs |
|---|---|---|
| Cleanlab | Python library that detects label errors using trained classifiers and confident learning. | quietset needs no model training and makes no task-specific assumptions. It filters by cross-run stability rather than estimated label quality. |
| Label Studio | Web-based annotation platform for labelling images, text, audio, and time series. | quietset is a CLI/library primitive, not an annotation UI. It measures stability of labels already produced by other tools. |
| pandas / polars | General-purpose data manipulation libraries. | quietset provides a purpose-built stability schema — decisions, per-dimension sub-scores, confidence, instability diagnostics — that would otherwise require substantial custom code. |
| Great Expectations / Soda | Data quality frameworks that validate data against rules (nulls, ranges, schema). | Those tools check whether data conforms to a schema. quietset checks whether labels or scores are consistent across repeated evaluations. |
| scipy.stats / sklearn metrics | Statistical functions such as Cohen's kappa and Fleiss' kappa. | quietset wraps similar ideas into a composable pipeline primitive with JSONL I/O, per-sample reports, confidence adjustment, and configurable thresholds. |
| LLM evaluation frameworks (RAGAS, DeepEval) | Frameworks that score LLM outputs against reference answers using model-based judges. | quietset is judge-agnostic. It takes whatever scores or labels your judges produce and measures agreement across runs, budgets, models, or seeds. |
Python bindings
crates/quietset-py provides Python bindings via pyo3 + maturin.
Status: alpha / experimental — the core scoring API is wrapped but the interface may change.
It is excluded from the Cargo workspace and from CI (cargo build/test --workspace never touches
it); build and test it manually from within crates/quietset-py.
&&
=
The bindings currently expose a single function, score_jsonl, which scores a JSONL string with
default settings and returns a JSONL string of results. The CLI is the stable interface with the
full set of commands and options; use Python bindings for embedding basic scoring in existing
Python pipelines where spawning a subprocess is impractical.
License
MIT OR Apache-2.0