Struct EntropyFilter 

pub struct EntropyFilter {
    pub max_entropy: f64,
    pub min_sources: usize,
    pub use_variance: bool,
    pub max_variance: f64,
}

Filter for rejecting high-entropy (conflicting) entity extractions.

Usage in RAG Systems

When multiple retrieval passes or models extract the same entity with different confidences, use this filter to:

1. Accept entities where sources agree (low entropy)
2. Flag/reject entities where sources disagree (high entropy)

Example

use anno_eval::eval::calibration::EntropyFilter;

let filter = EntropyFilter::new(0.6);  // Reject if entropy > 0.6

// Multiple models extracted "Apple" with these confidences:
let apple_scores = vec![0.95, 0.05, 0.5];  // Disagreement
assert!(!filter.should_keep(&apple_scores), "Should reject conflicting extractions");

let microsoft_scores = vec![0.9, 0.88, 0.92];  // Agreement
assert!(filter.should_keep(&microsoft_scores), "Should keep agreeing extractions");

Fields

max_entropy: f64

Maximum allowed entropy (0.0-1.0)

min_sources: usize

Minimum number of sources required

use_variance: bool

Use variance instead of entropy (faster, simpler)

max_variance: f64

Maximum variance threshold (if use_variance=true)

Implementations

impl EntropyFilter

pub fn new(max_entropy: f64) -> Self

Create with specific entropy threshold.

pub fn strict() -> Self

Create a strict filter (low threshold = high agreement required).

pub fn permissive() -> Self

Create a permissive filter (high threshold = accepts more disagreement).

pub fn should_keep(&self, scores: &[f64]) -> bool

Check if scores indicate sufficient agreement to keep the extraction.

pub fn compute_score(&self, scores: &[f64]) -> f64

Compute the entropy/variance for logging/debugging.

pub fn agreement_grade(&self, scores: &[f64]) -> &'static str

Grade the level of agreement.

Trait Implementations

impl Clone for EntropyFilter

fn clone(&self) -> EntropyFilter

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for EntropyFilter

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for EntropyFilter

fn default() -> Self

Returns the “default value” for a type.