roketok

A simple rust tokenization library, mostly
made for convenience so I, nor you have to
constantly write a new lexer/tokenizer
everytime you need to. Let's be real, it's really not thst fun to write the samething over again.

Currently roketok isn't aimed for being the best in the game, its far from that.
But it's meant to be lightweight and simple to use

TODO

• SIMD Support
• Token Trees
• Compile Time Tokenizer Generation