jayce 0.1.0

Jayce is blazing fast tokenizer
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44

# jayce


#### Token


```rust
pub struct Token {
    pub kind: String,
    pub value: String,
    pub line: usize,
    pub column: usize,
}
```

#### Sonant


```rust
Sonant::new(kind: &str, regex: &str);
```

#### Tokenizer


```rust
Jayce::new(source: String, sonants: Vec<Sonant>);
```

#### Example


```rust
let sonants: Vec<Sonant> = vec![
    Sonant::new("WhiteSpace", r"^\s+"),
    Sonant::new("keywords", "^(let|if|else)"),
    Sonant::new("identifiers", "^[a-z][a-z_]*"),
    Sonant::new("number", "^[0-9]+"),
    Sonant::new("operator", "^[-+*/%]"),
];

let source = "let math_exam = 89/100".to_owned();

let mut jayce = Jayce::new(source, sonants);

println!("{:?}", jayce.eat());

// Token { kind: "keywords", value: "let", line: 1, column: 4 }
```