Crate crossandra

Crossandra is a straightforward tokenization library designed for seamless text processing.

Examples

Brainfuck

use crossandra::Tokenizer;
use rustc_hash::FxHashMap;

let bf_tok = Tokenizer::default()
    .with_literals(&[
        ("add", "+"),
        ("sub", "-"),
        ("left", "<"),
        ("right", ">"),
        ("read", ","),
        ("write", "."),
        ("begin_loop", "["),
        ("end_loop", "]"),
    ])
    .expect("all literals should be ≥1 characters long");

for token in bf_tok.tokenize("cat program: ,[.,]").filter_map(Result::ok) {
   println!("{:?}", token);
}

Modules

common

A collection of common patterns for use in tokenizers.

Structs

Token

Represents a lexical token with a name/type and its raw value from the source code.

Tokenizer

The Crossandra tokenizer, operating on literals and patterns.

Type Aliases

FxHashMap

Type alias for a hash map that uses the Fx hashing algorithm.

FxHashSet

Type alias for a hash set that uses the Fx hashing algorithm.