Tokenizer-lib

Tokenization utilities for building parsers in Rust

Examples

Buffered token channel:

let mut stc = BufferedTokenQueue::new();
stc.push(Token(12, Span(0, 2)));
stc.push(Token(32, Span(2, 4)));
stc.push(Token(52, Span(4, 8)));
assert_eq!(stc.next().unwrap(), Token(12, Span(0, 2)));
assert_eq!(stc.next().unwrap(), Token(32, Span(2, 4)));
assert_eq!(stc.next().unwrap(), Token(52, Span(4, 8)));
assert_eq!(stc.next(), None);

(Multi-thread safe) Parallel token queue:

let (mut sender, mut reader) = ParallelTokenQueue::new();
std::thread::spawn(move || {
    sender.push(Token(12, Span(0, 2)));
    sender.push(Token(32, Span(2, 4)));
    sender.push(Token(52, Span(4, 8)));
});

assert_eq!(reader.next().unwrap(), Token(12, Span(0, 2)));
assert_eq!(reader.next().unwrap(), Token(32, Span(2, 4)));
assert_eq!(reader.next().unwrap(), Token(52, Span(4, 8)));
assert_eq!(reader.next(), None);

Generator token queue:

fn lexer(state: &mut u8, sender: &mut GeneratorTokenQueueBuffer<u8, ()>) {
    *state += 1;
    match state {
        1 | 2 | 3 => {
            sender.push(Token(*state * 2, ()))
        }
        _ => {}
    }
}

let mut reader = GeneratorTokenQueue::new(lexer, 0);

assert_eq!(reader.next().unwrap(), Token(2, ()));
assert_eq!(reader.next().unwrap(), Token(4, ()));
assert_eq!(reader.next().unwrap(), Token(6, ()));
assert!(reader.next().is_none());

Provides utilities such as peek and scan for lookahead. Also expect_next for expecting a token value.

tokenizer-lib 1.2.1

Tokenizer-lib

Examples