tinytoken
This library provides a tokenizer for parsing and categorizing different types of tokens, such as words, numbers, strings, characters, symbols, and operators. It includes configurable options to handle various tokenization rules and formats, enabling fine-grained control over how text input is parsed.
Example
use ;
Contributions
Feel free to send a PR to improve and/or extend the tool capabilities