jayce

Jayce is a simple tokenizer

Example

let mut jayce = jayce::Tokenizer::new();
jayce.add("newline", r"^\n");
jayce.add("whitespace", r"^\s+");
jayce.add("name", r"^[a-zA-Z_]+");
jayce.add("price", r"^[0-9]+\$");
jayce.add("equals", r"^=");

let source = "Excalibur = 5000$";

while let Some(token) = jayce.eat(source) {
    println!("{:?}", token);
}

Result

Token { kind: 2, start: 0, end: 9, line: 1, column: 10 }
Token { kind: 1, start: 9, end: 10, line: 1, column: 11 }
Token { kind: 4, start: 10, end: 11, line: 1, column: 12 }
Token { kind: 1, start: 11, end: 12, line: 1, column: 13 }
Token { kind: 3, start: 12, end: 17, line: 1, column: 18 }

Info

Reaching the end of source returns None

Unknown characters returns a Token with kind as None and value of the unknown character