Struct LabelShift 

pub struct LabelShift {
    pub overlap_ratio: f64,
    pub familiarity: f64,
    pub true_zero_shot_types: Vec<String>,
    pub transfer_difficulty: String,
}

Label shift between training and evaluation entity types.

Why This Matters

Imagine you trained a model on {PER, ORG, LOC} and then evaluate on {PERSON, COMPANY, CITY}. Is that zero-shot? Technically yes (new labels). Practically no (same concepts).

┌─────────────────────────────────────────────────────────────────────────┐
│                    THE LABEL SHIFT PROBLEM                              │
├─────────────────────────────────────────────────────────────────────────┤
│                                                                         │
│  TRAINING LABELS           EVAL LABELS         ARE THEY THE SAME?       │
│  ───────────────           ───────────         ──────────────────       │
│                                                                         │
│  PER ───────────────────── PERSON             ✓ Obviously (renamed)     │
│  ORG ───────────────────── COMPANY            ✓ Subset relationship     │
│  LOC ───────────────────── CITY               ✓ Subset relationship     │
│                                                                         │
│  ??? ←─────────────────── DISEASE            ✗ TRUE ZERO-SHOT!         │
│  ??? ←─────────────────── DRUG               ✗ TRUE ZERO-SHOT!         │
│                                                                         │
│  If 80% of eval types have training equivalents, your F1 is inflated.   │
└─────────────────────────────────────────────────────────────────────────┘

Embedding Space View

Labels that seem different can be close in embedding space:

                   EMBEDDING SPACE (2D projection)
                   ───────────────────────────────

           PER ●───────────────● PERSON
                     │
                very close in
               embedding space

           ORG ●─────● COMPANY

           LOC ●─────────● CITY


                                       ● DISEASE    ← Far from all
                                                      training types!
                                       ● DRUG       ← This is TRUE
                                                      zero-shot.

F1 on {PERSON, COMPANY, CITY}:  85%  (but model "knew" these)
F1 on {DISEASE, DRUG}:          45%  (honest zero-shot)

Research Context (arXiv:2412.10121 “Familiarity”)

Key findings from Golde et al. (2024):

• 80%+ label overlap in NuNER/PileNER → inflated F1 scores
• True zero-shot: evaluate only on types NOT in training
• Familiarity = semantic similarity × frequency weighting

Example

use anno_eval::eval::LabelShift;

let shift = LabelShift {
    overlap_ratio: 0.85,    // 85% of eval types in train
    familiarity: 0.72,      // Semantic similarity score
    true_zero_shot_types: vec!["DISEASE".into(), "DRUG".into()],
    transfer_difficulty: "low".into(),
};

// High overlap = easy transfer, but NOT true zero-shot
assert!(shift.is_inflated());

Fields

overlap_ratio: f64

Fraction of eval types found in training data (exact string match).

familiarity: f64

Familiarity score: semantic similarity weighted by frequency. Range: [0, 1]. Higher = more similar training/eval types.

true_zero_shot_types: Vec<String>

Entity types in eval NOT present in training (true zero-shot).

transfer_difficulty: String

Qualitative difficulty: “low”, “medium”, “high”.

Implementations

impl LabelShift

pub fn is_inflated(&self) -> bool

Check if F1 scores are likely inflated due to high label overlap.

Threshold from Familiarity paper: >0.8 overlap is concernoing.

pub fn true_zero_shot_count(&self) -> usize

Get count of true zero-shot types.

pub fn from_type_sets(train_types: &[String], eval_types: &[String]) -> Self

Compute label shift from training and eval type sets.

Arguments

• train_types - Entity types seen during training
• eval_types - Entity types in evaluation benchmark

Note

This computes both string-match overlap and semantic similarity-based familiarity. For true semantic similarity, use from_type_sets_with_embeddings() if embeddings are available. See arXiv:2412.10121 for details.

pub fn from_type_sets_with_embeddings<F>( train_types: &[String], eval_types: &[String], embedding_fn: F, ) -> Self

where F: Fn(&str) -> Option<Vec<f32>>,

Compute label shift with embedding-based familiarity.

Arguments

• train_types - Entity types seen during training
• eval_types - Entity types in evaluation benchmark
• embedding_fn - Function that computes embedding for a label name

Note

This computes true semantic similarity using embeddings, as recommended in the Familiarity paper (arXiv:2412.10121). Familiarity = semantic similarity × frequency weighting.

The embedding function should return a normalized vector (unit length) for cosine similarity.

Trait Implementations

impl Clone for LabelShift

fn clone(&self) -> LabelShift

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for LabelShift

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl<'de> Deserialize<'de> for LabelShift

fn deserialize<__D>(__deserializer: __D) -> Result<Self, __D::Error>

where __D: Deserializer<'de>,

Deserialize this value from the given Serde deserializer.

impl Display for LabelShift

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Serialize for LabelShift

fn serialize<__S>(&self, __serializer: __S) -> Result<__S::Ok, __S::Error>

where __S: Serializer,

Serialize this value into the given Serde serializer.