Expand description
Token types returned by the segmenter.
Structs§
- Token
- A single token produced by
crate::Tokenizer::segment.
Enums§
- Named
Entity Kind - Named entity category assigned by the NE gazetteer.
- Token
Kind - Classification of a
Token’s script / category.