Struct TaskEvaluator 

pub struct TaskEvaluator { /* private fields */ }

Evaluator for task-dataset-backend combinations.

Implementations

impl TaskEvaluator

pub fn history(&self) -> Option<&EvalHistory>

Access the evaluation history (if initialized).

pub fn is_task_supported(task: Task) -> bool

True if this task has a real end-to-end evaluation path implemented in TaskEvaluator.

Tasks may be “catalogued” (present in the dataset registry and task mapping) without having an evaluation pipeline yet; those tasks should not be scheduled by the matrix.

pub fn new() -> Result<Self>

Create a new task evaluator.

pub fn with_cache_dir(cache_dir: impl AsRef<Path>) -> Result<Self>

Create a new task evaluator with a custom cache directory.

Useful for testing with isolated caches.

pub fn evaluate_all( &self, config: TaskEvalConfig, ) -> Result<ComprehensiveEvalResults>

Run comprehensive evaluation across all valid combinations.

pub fn map_dataset_labels_to_model( dataset_labels: &[&str], backend_name: &str, ) -> Vec<String>

Map dataset entity type labels to model-compatible labels.

Handles common label variations (e.g., “PER” → “person”, “PERSON” → “person”). Also handles domain-specific mappings (e.g., MIT Movie “Actor” → “person”). Also limits labels for backends with restrictions (e.g., NuNER only supports 3 labels). Public for testing purposes.

impl TaskEvaluator

pub fn compute_robustness( &self, backend_name: &str, dataset_data: &LoadedDataset, config: &TaskEvalConfig, ) -> Option<RobustnessResults>

Compute robustness testing results.

Performance Note

This function creates a new backend instance and runs robustness tests on up to ROBUSTNESS_TEST_LIMIT examples. This is intentional - robustness testing requires running perturbations that may affect backend state.

Limitations

• Limited to ROBUSTNESS_TEST_LIMIT examples for performance
• Creates a new backend instance (doesn’t reuse from main evaluation)

pub fn compute_stratified_metrics( &self, dataset_data: &LoadedDataset, metrics: &HashMap<String, f64>, ) -> Option<StratifiedMetrics>

Compute stratified metrics across multiple dimensions.

Fallback Behavior

This is a fallback when per-example predictions are not available. All entity types will show the same aggregate F1 metrics because we lack the per-prediction data needed for true per-type stratification.

Preferred Path

For proper per-type stratification, use Self::compute_stratified_metrics_from_scores which computes actual per-type F1/precision/recall from per-example predictions. That method is automatically used when per-example scores are available via the evaluation pipeline (see evaluate_ner_internal).

When This Fallback Is Used

• External evaluation without per-example tracking
• Legacy integrations that only provide aggregate metrics
• Quick estimates when full stratification isn’t needed

Trait Implementations

impl Default for TaskEvaluator

fn default() -> Self

Creates a default TaskEvaluator.

Panics

This function will panic if DatasetLoader::new() fails. In production code, prefer using TaskEvaluator::new() which returns a Result.