json_five

Module tokenize

Source
Expand description

The default performance-focused tokenizer

Structs§

TokenizationError
TokenizerConfig
Tokens

Enums§

TokType

Functions§

tokenize_bytes
Tokenize bytes to Tokens
tokenize_rt_bytes
tokenize_rt_str
tokenize_str
Turn str into Tokens.