Expand description
module for Token type
Structs§
- Token
- a reference counted structure with a token and its id
- Token
Into Iter - a simple dictionary based tokenizer dictionary where single bytes implicitly form the lower 256 ids. It’s reference counted and cheap to clone