Skip to main content

Module scoring

Module scoring 

Source
Expand description

100-point model quality scoring system (spec §7) 100-Point Model Quality Scoring System (spec §7)

Evaluates models across seven dimensions based on data science and ML best practices, aligned with Toyota Way principles:

DimensionMax PointsToyota Way Principle
Accuracy & Performance25Kaizen (continuous improvement)
Generalization & Robustness20Jidoka (quality built-in)
Model Complexity15Muda elimination (waste reduction)
Documentation & Provenance15Genchi Genbutsu (go and see)
Reproducibility15Standardization
Security & Safety10Poka-yoke (error-proofing)

§References

  • [Raschka 2018] Model Evaluation, Model Selection, and Algorithm Selection in ML
  • [Hastie et al. 2009] The Elements of Statistical Learning
  • [Mitchell et al. 2019] Model Cards for Model Reporting
  • [Gebru et al. 2021] Datasheets for Datasets
  • [Pineau et al. 2021] ML Reproducibility Checklist

Structs§

CriticalIssue
Critical issue that must be addressed
DimensionScore
Score for a single dimension
DimensionScores
Individual dimension scores
ModelFlags
Model feature flags
ModelMetadata
Model metadata for scoring
QualityScore
100-point model quality score
ScoreBreakdown
Breakdown item for dimension scoring
ScoringConfig
Configuration for quality scoring
TrainingInfo
Training information

Enums§

Finding
Finding from quality analysis
Grade
Letter grade for quality score
ScoredModelType
Model type for scoring context
Severity
Critical issue severity

Functions§

compute_quality_score
Compute quality score for a model