Struct StatisticalFilterConfig 

pub struct StatisticalFilterConfig {
    pub compression_ratio: f32,
    pub idf_weight: f32,
    pub position_weight: f32,
    pub pos_weight: f32,
    pub entity_weight: f32,
    pub entropy_weight: f32,
    pub enable_protection_masks: bool,
    pub enable_contextual_stopwords: bool,
    pub preserve_negations: bool,
    pub preserve_comparators: bool,
    pub domain_terms: Vec<String>,
    pub min_gap_between_critical: usize,
}

Configuration for statistical filtering

Fields

compression_ratio: f32

Target compression ratio (0.0 to 1.0) 0.5 = keep 50% of tokens, 0.2 = keep 20%

idf_weight: f32

Weight for inverse document frequency (IDF)

position_weight: f32

Weight for position in document (start/end more important)

pos_weight: f32

Weight for part-of-speech heuristics

entity_weight: f32

Weight for named entity patterns

entropy_weight: f32

Weight for local entropy (vocabulary diversity)

enable_protection_masks: bool

Enable protection masks for code/JSON/paths/identifiers

enable_contextual_stopwords: bool

Enable contextual stopword filtering (smarter removal)

preserve_negations: bool

Preserve negations (not, no, never, don’t, etc.)

preserve_comparators: bool

Preserve comparators (!=, <=, >=, ==, etc.)

domain_terms: Vec<String>

Domain-specific terms to always preserve

min_gap_between_critical: usize

Minimum gap between critical tokens before re-adding

Trait Implementations

impl Clone for StatisticalFilterConfig

fn clone(&self) -> StatisticalFilterConfig

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for StatisticalFilterConfig

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for StatisticalFilterConfig

fn default() -> Self

Returns the “default value” for a type.