Modules§
- config
- Provides the configurations for tokenizers the most basic being:
- prelude
- Gives you all the basic utilities without scavenging for them.
- record
- Provides
Recordwhich records the tokenizers data, allowing for more complex tokenizations. - tokens
- Provides the essentials for interacting and manipulating tokens after tokenization.
Structs§
- Tokenizer
- Tokenizer