Module tokenize

Source
Expand description

The default performance-focused tokenizer

Structs§

TokenizationError
TokenizerConfig
Tokens

Enums§

TokType

Functions§

tokenize_bytes
Like tokenize_str but for bytes
tokenize_rt_bytes
Like tokenize_rt_str but for bytes
tokenize_rt_str
Like tokenize_str but includes whitespace and comment tokens
tokenize_str
Turn str into Tokens.