Module tokenizer 

Tokenization of a string.

• Tokenizer - Performs the tokenization.
• TokenizerOptions - Various configurations for the Tokenizer for example alternative symbols for some tokens.
• Token - A struct that describes a token - it contains the text of the token along with its type.
• TokenType - The type of the token.
• TokenizerError - The error that the Tokenizer will throw in case of an error.

Example

use mini_builder_rs::tokenizer::{Tokenizer, TokenizerOptions};

let source = "is 1 bigger than 2? {{ 1 > 2 ? 'yes' : 'no' }}";
let tokens = Tokenizer::new(source, TokenizerOptions::default())
    .tokenize()
    .unwrap();
// tokens will be a vector of `Token<'a>`s where `'a` is the lifetime of
// `source`

Modules

token

tokenizer_error

Structs

Tokenizer

Given a string slice, tokenizes it to a vector of Tokens.

TokenizerOptions

Used to configure the Tokenizer.