roketok 0.3.1

This crate provides a way to simply set up a tokenizer and use it. Not recommended for simple tokenizers as this crate adds a bunch of stuff to support many if not all kinds of tokenizers.
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88

# roketok

> [!WARNING]
> **roketok** is still under constant shifts \
> and changes, though finalisation is near.

A simple tokenization library, focused on ease of use.

If you find an issue, whether is performance or just bugs in general, please submit an issue in [issues](https://github.com/rok3tt/roketok/issues).

## Basic C Tokenization Example
```rust
use roketok::prelude::*;

#[derive(Default)]
enum TokenKind {
    Identifier,
    Number,
    
    Asterisk,
    Ampersand,
    Semicolon,
    
    Equal,
    AddEqual,
    
    Parenthesis,
    
    #[default]
    Invalid,
}

fn main() {
    let contents = r#"
        void foo(int *value) {
            *value += 35;
        }
        
        int main(void) {
            int value = 34;
            foo(&value);
            return value;
        }
    "#;
    
    let config = Configuration::new()
        .add_tokens([
            (TokenConfiguration::Rule(&|iter, _| {
                if let Some(char) = iter.last() {
                    if !char.is_alphabetic() { return false; }
                    while let Some(char) = iter.peek() {
                        if !char.is_alphanumeric() { break; }
                        let _ = iter.next();
                    }
                    return true;
                }
                false
            }), TokenKind::Identifier),
            (TokenConfiguration::Rule(&|iter, _| {
                if let Some(char) = iter.last() {
                    if !char.is_numeric() { return false; }
                    while let Some(char) = iter.peek() {
                        if !char.is_alphanumeric() { break; }
                        let _ = iter.next();
                    }
                    return true;
                }
                false
            }), TokenKind::Number),
            
            (TokenConfiguration::Boring(&['*']), TokenKind::Asterisk),
            (TokenConfiguration::Boring(&['&']), TokenKind::Ampersand),
            
            (TokenConfiguration::Boring(&['=']), TokenKind::Equal),
            (TokenConfiguration::Boring(&['+', '=']), TokenKind::AddEqual),
            
            (TokenConfiguration::Boring(&[';']), TokenKind::Semicolon),
            
            (TokenConfiguration::Branch(&['('], &[')']), TokenKind::Parenthesis),
        ]);
    let mut tokenizer = Tokenizer::new(&config, contents);
    let tree = tokenizer.build();
}
```

### TODO
- [ ] SIMD Support
- [ ] Compile Time Tokenizer Generation