oxibonsai-tokenizer-0.1.1
Platform
Feature flags

docs.rs

Rust

oxibonsai-tokenizer 0.1.1

Pure Rust BPE tokenizer for OxiBonsai (MeCrab-compatible)

Crate
Source
Builds
Feature flags

..
bpe.rs
error.rs
lib.rs
serialization.rs
tests.rs
tokenizer.rs
trainer.rs
utils.rs
vocab.rs