roketok 0.2.1

This crate makes it easy for anyone to setup tokenization. It was primarily created because I got sick of writing tokenizers every time I wanted to parse something, so a lot of this is just for my own will but use this as you'd like.

Documentation

Coverage
66.67%
10 out of 15 items documented2 out of 8 items with examples
Size
Source code size: 12.58 kB This is the summed size of all the files inside the crates.io package for this release.
Documentation size: 1.92 MB This is the summed size of all files generated by rustdoc for all configured targets
Ø build duration
this release: 12s Average build duration of successful builds.
all releases: 15s Average build duration of successful builds in releases after 2024-10-23.
Links
Repository
crates.io
Dependencies
Versions
Owners

roketok

[!WARNING] You must use version >=0.2.1 as older crates have a
breaking bug that will cause issues on most use cases.

A simple tokenization library, focused on ease of use. Currently roketok is very much still under plenty of changes and currently is limited to simple linear tokenization.

If you find an issue, whether is performance or just bugs in general, please submit an issue in issues.

TODO

SIMD Support
Token Trees
Compile Time Tokenizer Generation