Struct TaskEvalConfig 

pub struct TaskEvalConfig {

    pub tasks: Vec<Task>,
    pub datasets: Vec<DatasetId>,
    pub backends: Vec<String>,
    pub max_examples: Option<usize>,
    pub seed: Option<u64>,
    pub require_cached: bool,
    pub relation_threshold: f32,
    pub robustness: bool,
    pub compute_familiarity: bool,
    pub temporal_stratification: bool,
    pub confidence_intervals: bool,
    pub custom_coref_resolver: Option<Arc<dyn CoreferenceResolver>>,
    pub coref_use_gold_mentions: bool,
}

Configuration for task evaluation.

Fields

tasks: Vec<Task>

Which tasks to evaluate

datasets: Vec<DatasetId>

Which datasets to use (if empty, uses all suitable datasets for each task)

backends: Vec<String>

Which backends to test (if empty, uses all compatible backends)

max_examples: Option<usize>

Maximum number of examples per dataset (for quick testing)

seed: Option<u64>

Random seed for sampling (for reproducibility and varied testing)

require_cached: bool

Whether to skip datasets that aren’t cached

relation_threshold: f32

Confidence threshold for relation extraction (default: 0.5)

robustness: bool

Whether to run robustness testing (perturbations)

compute_familiarity: bool

Whether to compute familiarity scores for zero-shot evaluations

temporal_stratification: bool

Whether to compute temporal stratification (if dataset supports it)

confidence_intervals: bool

Whether to compute confidence intervals for metrics

custom_coref_resolver: Option<Arc<dyn CoreferenceResolver>>

Optional custom coreference resolver (for use with matryoshka-box trained models) If None, resolver is created from backend_name using create_coref_resolver() Uses Arc to allow sharing across multiple evaluation calls

coref_use_gold_mentions: bool

Coreference evaluation mode:

• false (default): run NER to get mentions, then resolve coref on those mentions.
• true: use GOLD mentions from the coref dataset and evaluate clustering only.

This is critical for datasets like CorefUD where mentions include pronouns/nominals and empty nodes (zero anaphora) that typical NER backends do not emit.

Trait Implementations

impl Debug for TaskEvalConfig

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for TaskEvalConfig

fn default() -> Self

Returns the “default value” for a type.

impl<'de> Deserialize<'de> for TaskEvalConfig

fn deserialize<__D>(__deserializer: __D) -> Result<Self, __D::Error>

where __D: Deserializer<'de>,

Deserialize this value from the given Serde deserializer.

impl Serialize for TaskEvalConfig

fn serialize<__S>(&self, __serializer: __S) -> Result<__S::Ok, __S::Error>

where __S: Serializer,

Serialize this value into the given Serde serializer.