Struct TextUtils 

pub struct TextUtils;

Text processing utilities

Implementations

impl TextUtils

pub fn tokenize(text: &str) -> Vec<String>

Simple whitespace tokenization

pub fn count_stopwords(text: &str, stopwords: &HashSet<String>) -> usize

Count stopwords in text

pub fn stopword_density(text: &str, stopwords: &HashSet<String>) -> f32

Calculate stopword density

pub fn split_sentences(text: &str) -> Vec<String>

Split text into sentences

pub fn token_f1( extracted: &str, reference: &str, stopwords: &HashSet<String>, ) -> f32

Token-level F1 between an extracted text and a reference (ground truth).

Tokens are lowercased, stopword-filtered and required to be longer than two characters — matching GroundTruthEvaluator’s normalization so the training reward and the offline evaluation metric agree. Returns a value in [0, 1]. If the reference is empty the result is 0 (caller decides on a fallback reward).

pub fn calculate_text_quality(text: &str, stopwords: &HashSet<String>) -> f32

Calculate text quality score