Struct Tokenizer 

pub struct Tokenizer<'a> { /* private fields */ }

The Crossandra tokenizer, operating on literals and patterns.

Literals

Literals indicate values that have to be exactly matched by the tokenizer. They are represented by a slice of (name, value) pairs. For example, a literal map for Brainfuck would be defined like this:

let literals = [
    ("add", "+"),
    ("sub", "-"),
    ("left", "<"),
    ("right", ">"),
    ("read", ","),
    ("write", "."),
    ("begin_loop", "["),
    ("end_loop", "]"),
];

Literals take precedence over patterns.

Patterns

Patterns are regular expressions that match more complex token structures. They are represented as pairs of strings (name, pattern) in a Vec to maintain a consistent matching order.

The order of patterns matters as the tokenizer will use the first matching pattern it finds. Duplicate pattern names are not allowed and will result in an error. This crate also provides a collection of commonly used patterns in the common module. For example, patterns covering binary, octal, and hexadecimal literals could be defined like this:

let patterns = vec![
    ("binary".into(), r"0[bB][01]+".into()),
    ("octal".into(), r"0[Oo][0-7]+".into()),
    ("hexadecimal".into(), r"(?i)0x[0-9a-f]+".into()),
];

Other options

ignore_whitespace

Whether to ignore the following whitespace characters:

Code	Character
0x9	Tab (\t)
0xa	Line feed (\n)
0xb	Vertical tab
0xc	Form feed
0xd	Carriage return (\r)
0x20	Space ( )

Defaults to false.

ignored_characters

A set of characters to ignore during tokenization. Defaults to an empty Vec.

Fast Mode

When all literals are of length 1 and there are no patterns, Crossandra uses a simpler tokenization method.

For instance, tokenizing a 1MB random Brainfuck file with 10% of the file being comments is ~300x faster with Fast Mode (32.5s vs 110ms on Apple M2).

Do note that this is a rather extreme case; for a 1KB file, the speedup is ~2.3x.

Implementations

impl<'a> Tokenizer<'a>

pub fn new( literals: &[(&'a str, &'a str)], patterns: Vec<(String, String)>, ignored_characters: FxHashSet<char>, ignore_whitespace: bool, ) -> Result<Self, Error>

Creates a new Tokenizer with the specified configuration.

Errors

This function will return an error if:

• any literal is empty,
• there are duplicate patterns, or
• any pattern regex is invalid.

pub fn tokenize( &'a self, source: &'a str, ) -> Box<dyn Iterator<Item = Result<Token, Error>> + 'a>

Tokenizes the given source code and returns an Iterator of Tokens.

pub fn tokenize_lines( &'a self, source: &'a str, ) -> impl ParallelIterator<Item = Result<Vec<Token>, Error>> + 'a

Splits the given source code into lines and tokenizes each line separately. Returns an Iterator of Vecs of Tokens.

Errors

This function will return an error if any line fails to tokenize.

pub fn with_literals( self, literals: &[(&'a str, &'a str)], ) -> Result<Self, Error>

Sets the literals of this Tokenizer and returns itself.

Errors

This function will return an error if any literal is empty.

pub fn with_patterns( self, patterns: Vec<(String, String)>, ) -> Result<Self, Error>

Sets the patterns of this Tokenizer and returns itself.

Errors

This function will return an error if:

• there are duplicate patterns, or
• any pattern regex is invalid.

pub fn with_ignored_characters( self, ignored_characters: FxHashSet<char>, ) -> Self

Sets the ignored characters of this Tokenizer and returns itself.

pub fn with_ignore_whitespace(self, ignore_whitespace: bool) -> Self

Sets the ignore_whitespace option of this Tokenizer and returns itself.

pub fn set_literals( &mut self, literals: &[(&'a str, &'a str)], ) -> Result<(), Error>

Sets the literals of this Tokenizer.

Errors

This function will return an error if any literal is empty.

pub fn set_patterns( &mut self, patterns: Vec<(String, String)>, ) -> Result<(), Error>

Sets the patterns of this Tokenizer.

Errors

This function will return an error if:

• there are duplicate patterns, or
• any pattern regex is invalid.

pub fn set_ignored_characters(&mut self, ignored_characters: FxHashSet<char>)

Sets the ignored characters of this Tokenizer.

pub fn set_ignore_whitespace(&mut self, ignore_whitespace: bool)

Sets the ignore_whitespace option of this Tokenizer.

Trait Implementations

impl<'a> Clone for Tokenizer<'a>

fn clone(&self) -> Tokenizer<'a>

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl<'a> Debug for Tokenizer<'a>

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for Tokenizer<'_>

fn default() -> Self

Returns the “default value” for a type.

impl PartialEq for Tokenizer<'_>

fn eq(&self, other: &Self) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl Eq for Tokenizer<'_>