jayce 4.0.0

jayce is a simple tokenizer 🌌
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41

# jayce

Jayce is a simple tokenizer

##### Example

```rust
use jayce::{Tokenizer, TokenizerResult};

let source = "Excalibur = 5000$";
let duos = vec![
    ("newline", r"^\n"),
    ("whitespace", r"^\s+"),
    ("name", r"^[a-zA-Z_]+"),
    ("price", r"^[0-9]+\$"),
    ("equals", r"^="),
];
let mut jayce = Tokenizer::new(source, &duos);

while let TokenizerResult::Token(token) = jayce.next() {
    println!("{:?}", token);
}
```

##### Result

```rust,ignore
Token { kind: "name", value: "Excalibur", line: 1, column: 10 }
Token { kind: "whitespace", value: " ", line: 1, column: 11 }
Token { kind: "equals", value: "=", line: 1, column: 12 }
Token { kind: "whitespace", value: " ", line: 1, column: 13 }
Token { kind: "price", value: "5000$", line: 1, column: 18 }
```

##### Info

`next` returns a `TokenizerResult` which can be

1. `Found(token)` If a regex matches
2. `Unknown(message)` When there is zero match
3. `End` Reaching the source ends