Expand description
Compound token splitter using the word lexicon. Splits a compound token into valid sub-word pairs using the lexicon.
Structs§
- Token
Splitter - Splits compound Persian tokens into their constituent words.
Compound token splitter using the word lexicon. Splits a compound token into valid sub-word pairs using the lexicon.