Expand description
Things that add more metadata to tokens.
Sometimes these are prerequisites for other segmenters/normalizers.
Keep in mind that the order in the chain where they are inserted matters.
Structs§
- Augmentation
Classify - An augmenter that rewrites the SegmentedToken::kind field to match reality.
- Augmentation
Detect Language - Will run language and script detection using whatlang.
- Augmentation
Detect Script - Will run just script detection using whatlang.
Traits§
- Augmenter
- Simplified Interface that assumes one token in, one token out.