Skip to main content

norm_token

Function norm_token 

Source
pub fn norm_token(t: &str) -> String
Expand description

Light, deterministic singular form of a (lowercase) token, so the surface-form channels — keyword, phrase, BM25 — match across trivial inflection (“spreadsheets” ↔ “spreadsheet”, “dependencies” ↔ “dependency”). Not a real stemmer: it only needs to be consistent, because both the prompt side and the skill side are normalized through it at match time. Applied at match time only — never inside the embedders — so persisted index vectors are untouched.