Module utils 

Helpers for the visualizer: a simple tokenizer, metrics and issue detection.

Structs

AnalysisMetrics

Calculate various metrics from token analysis

Functions

create_mock_analysisDeprecated

Made-up confidence scores (a hash of each word), for exercising the renderers without model data. The numbers mean nothing.

detect_issues

Detect potential issues in token analysis

simple_tokenize

Simple tokenizer for demo purposes - splits on whitespace and punctuation