Crate tinytoken

tinytoken

This library provides a tokenizer for parsing and categorizing different types of tokens, such as words, numbers, strings, characters, symbols, and operators. It includes configurable options to handle various tokenization rules and formats, enabling fine-grained control over how text input is parsed.

Example

use tinytoken::{Tokenizer, TokenizerBuilder, Choice};

fn main() {
    let tokenizer = TokenizerBuilder::new()
        .parse_char_as_string(true)
        .allow_digit_separator(Choice::Yes('_'))
        .add_symbol('$')
        .add_operators(&['+', '-'])
        .build("let x = 123_456 + 0xFF");

    match tokenizer.tokenize() {
        Ok(tokens) => {
            for token in tokens {
                println!("{:?}", token);
            }
        }
        Err(err) => {
            eprintln!("Tokenization error: {err}");
        }
    }
}

Contributions

Feel free to send a PR to improve and/or extend the tool capabilities

Modules

error

Contains error definitions specific to tokenization

Structs

Loc

Represents the location of a token in the input text, with line and column values

Token

Represents an individual token with type, value, and location

Tokenizer

Primary struct for tokenizing an input string, with methods for parsing and generating tokens

TokenizerBuilder

A builder struct for creating a TokenizerConfig instance with customized options

TokenizerConfig

Configuration struct for the tokenizer, allowing customization of tokenization behavior

Enums

Choice

Configurable option for specific settings in TokenizerConfig

NumberType

Represents the types of numeric tokens recognized by the tokenizer

TokenType

Represents all possible token types that can be parsed by the tokenizer