jayce 2.0.0

jayce is a simple tokenizer 🌌
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44

# jayce


Jayce is a simple tokenizer

#### Example


```rust
use jayce::Tokenizer;

fn main() {
    let source = "let value = 5000$";
    let duos = &[
        ("keywords", r"^(let|const)"),
        ("whitespace", r"^\s+"),
        ("variable", r"^[a-zA-Z_]+"),
        ("price", r"^[0-9]+\$"),
        ("equals", r"^="),
        ("newline", r"^\n"),
    ];
    let mut tokenizer = Tokenizer::new(source, duos);

    while let Some(token) = tokenizer.eat() {
        println!("{:?}", token);
    }
}
```

#### Result


```rust,ignore
Token { kind: Some("keywords"), value: "let", line: 1, column: 4 }
Token { kind: Some("whitespace"), value: " ", line: 1, column: 5 }
Token { kind: Some("variable"), value: "value", line: 1, column: 10 }
Token { kind: Some("whitespace"), value: " ", line: 1, column: 11 }
Token { kind: Some("equals"), value: "=", line: 1, column: 12 }
Token { kind: Some("whitespace"), value: " ", line: 1, column: 13 }
Token { kind: Some("price"), value: "5000$", line: 1, column: 18 }
```

#### Info


Reaching the end of source returns `None`

Unknown characters returns a `Token` with `kind` as `None` and `value` of the unknown character