liendl_tokenizer 0.1.0

A simple BPE tokenizer for Rust
Documentation
  • Coverage
  • 0%
    0 out of 69 items documented0 out of 53 items with examples
  • Size
  • Source code size: 30.51 kB This is the summed size of all the files inside the crates.io package for this release.
  • Documentation size: 1.24 MB This is the summed size of all files generated by rustdoc for all configured targets
  • Ø build duration
  • this release: 21s Average build duration of successful builds.
  • all releases: 21s Average build duration of successful builds in releases after 2024-10-23.
  • Links
  • crates.io
  • Dependencies
  • Versions
  • Owners
  • jonasliendl

BPE-Tokenizer

This is a simple BPE-Tokenizer I've created for the electoral module Foundational Generative Models at my university. I will document everything here in this repo and on the website I'm going to build where you can also test the tokenizer on different versions with different methods.

🛣️ Roadmap

Status Feature
✅ Text Loader supporting .TXT files
✅ Training process function
✅ Tokenize text function
✅ Export function for vocabulary
🕣 Website to show off tokenizer
✅ Support Case-Sensitivity
✅ Support UTF-8 text
☑️ Handle unknown characters using the Out-of-Vocabulary method
🕣 Train Tokenizer on English and German text
✅ Add token decoding
☑️ Support additional languages like Durch, Spanish, Polish, French, etc.
☑️ Support additional file formats like CSV or Excel for Text Loader
✅ Convert to library
🕣 Include ability to add special tokens