roketok 0.1.4

This crate makes it easy for anyone to setup tokenization. It was primarily created because I got sick of writing tokenizers every time I wanted to parse something, so a lot of this is just for my own will but use this as you'd like.
Documentation
roketok-0.1.4 has been yanked.

roketok

A simple rust tokenization library, mostly
made for convenience so I, nor you have to
constantly write a new lexer/tokenizer
everytime you need to. Let's be real, it's really not thst fun to write the samething over again.

Currently roketok isn't aimed for being the best in the game, its far from that.
But it's meant to be lightweight and simple to use

TODO

  • SIMD Support
  • Token Trees
  • Compile Time Tokenizer Generation