Module task_evaluator 

Comprehensive Task-Dataset-Backend Evaluation System

This module provides a unified evaluation framework that:

• Maps tasks to suitable datasets
• Maps datasets to compatible backends
• Runs evaluations across all valid combinations
• Generates comprehensive reports

Design Philosophy

• Trait-based: Backend capabilities detected via trait implementations
• Many-to-many: Each task can use multiple datasets, each dataset can evaluate multiple tasks
• Comprehensive: Evaluates all valid task-dataset-backend combinations
• Extensible: Easy to add new tasks, datasets, or backends

Structs

ComprehensiveEvalResults

Comprehensive evaluation results across all combinations.

ConfidenceIntervals

Confidence intervals for key metrics.

EvalSummary

Summary statistics for comprehensive evaluation.

MetricWithCI

Metrics with confidence intervals.

StratifiedMetrics

Stratified metrics across multiple dimensions.

TaskEvalConfig

Configuration for task evaluation.

TaskEvalResult

Results from evaluating a task-dataset-backend combination.

TaskEvaluator

Evaluator for task-dataset-backend combinations.