kcode-speaker-eval 0.1.0

Deterministic leakage-safe offline evaluation for Kennedy speaker models
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78

# kcode-speaker-eval 0.1.0

Deterministic, leakage-safe offline evaluation and configuration selection for the frozen 24-field speaker-classification libraries.

## API

```rust
pub struct EvaluationPlan<'a> {
    pub dataset: &'a Dataset,
    pub tuning_folds: &'a [OpenSetFold],
    pub held_out_fold: &'a OpenSetFold,
    pub candidates: &'a [ModelConfig],
}

pub struct AggregateMetrics {
    pub known_total: usize,
    pub known_correct: usize,
    pub known_misidentified: usize,
    pub known_rejected: usize,
    pub unknown_total: usize,
    pub unknown_rejected: usize,
    pub unknown_accepted: usize,
}

impl AggregateMetrics {
    pub fn known_identification_rate(&self) -> f64;
    pub fn known_misidentification_rate(&self) -> f64;
    pub fn known_rejection_rate(&self) -> f64;
    pub fn unknown_rejection_rate(&self) -> f64;
    pub fn unknown_acceptance_rate(&self) -> f64;
    pub fn balanced_open_set_rate(&self) -> f64;
}

pub struct CandidateEvaluation {
    pub config: ModelConfig,
    pub tuning: AggregateMetrics,
}

pub struct EvaluationResult {
    pub selected_config: ModelConfig,
    pub candidate_evaluations: Vec<CandidateEvaluation>,
    pub held_out: AggregateMetrics,
}

pub enum EvalError {
    EmptyCandidates,
    EmptyTuningFolds,
    DuplicateCandidate,
    DuplicateTuningFold,
    InvalidCandidate,
    InvalidFold,
    FoldOutsideOuterTraining,
    GroupLeakage,
    OpenSetViolation,
    CountOverflow,
    Model(ModelError),
}

pub fn evaluate(plan: EvaluationPlan<'_>) -> Result<EvaluationResult, EvalError>;
```

`Dataset`, `OpenSetFold`, `ModelConfig`, and `ModelError` are the exact types from `kcode-speaker-dataset` 0.2.0 and `kcode-speaker-model` 0.2.0.

## Plan boundaries

The held-out fold must partition every active dataset row exactly once into nonempty training, known-test, and pseudo-unknown-test sets. Every tuning fold must similarly partition exactly the held-out training rows. Indices must be in range and unique within a fold.

For every fold, a `group_id` stays wholly on the training side or wholly on the test side. The pseudo-unknown speaker has no training row, all of that speaker's rows are unknown test rows, co-speakers from the same groups remain test-only, and every known-test speaker has training evidence. Invalid, duplicate, incomplete, overlapping, out-of-universe, or open-set-unsafe plans fail closed.

## Evaluation and selection

Each candidate is fit independently on each tuning fold's training rows. Normalization and all model statistics are therefore learned only from that partition. Tuning rows are scored only after fitting. Candidate settings include model settings and both thresholds.

Candidates and tuning folds are canonicalized independently of caller order. Selection first maximizes the mean of known-speaker correct-identification rate and unknown-speaker rejection rate, then known-identification rate, then unknown-rejection rate, then minimizes known misidentifications, and finally chooses the lexicographically smallest configuration by mask bits, component count, relevance, variance floor, absolute threshold, and margin threshold using total floating-point order. Duplicate or non-finite configurations are rejected.

After selection, the chosen configuration is fit once on the complete held-out training partition. Each held-out known and unknown row is then scored exactly once. `AggregateMetrics` exposes counts and finite rates for known correct identification, known misidentification, known rejection, unknown rejection, and unknown acceptance. A rate with a zero denominator is defined as `0.0`, although valid evaluation folds require both classes.

The package uses the exact-pinned 0.2.0 speaker types, dataset, and model contracts and therefore the frozen 24-feature schema. It performs no audio work, provider calls, storage, persistence, inventory construction, replay, corpus processing, host integration, activation, publication, or deployment.