Docs.rs
tokenizers-0.15.2
tokenizers 0.15.2
Docs.rs crate page
Apache-2.0
Links
Homepage
Documentation
Repository
crates.io
Source
Owners
n1t0
Narsil
ArthurZucker
Dependencies
aho-corasick ^1.1
normal
clap ^4.4
normal
derive_builder ^0.12
normal
esaxx-rs ^0.1.10
normal
fancy-regex ^0.13
normal
getrandom ^0.2.10
normal
hf-hub ^0.3.2
normal
indicatif ^0.17
normal
itertools ^0.12
normal
lazy_static ^1.4
normal
log ^0.4
normal
macro_rules_attribute ^0.2.0
normal
monostate ^0.1.9
normal
onig ^6.4
normal
paste ^1.0.14
normal
rand ^0.8
normal
rayon ^1.8
normal
rayon-cond ^0.3
normal
regex ^1.9
normal
regex-syntax ^0.8
normal
serde ^1.0
normal
serde_json ^1.0
normal
spm_precompiled ^0.1
normal
thiserror ^1.0.49
normal
unicode-normalization-alignments ^0.1
normal
unicode-segmentation ^1.10
normal
unicode_categories ^0.1
normal
assert_approx_eq ^1.1
dev
criterion ^0.5
dev
tempfile ^3.8
dev
Versions
47.09%
of the crate is documented
Go to latest version
Platform
x86_64-unknown-linux-gnu
Feature flags
Rust
About docs.rs
Privacy policy
Rust website
The Book
Standard Library API Reference
Rust by Example
The Cargo Guide
Clippy Documentation
tokenizers
0.15.2
Module models
Modules
Enums
In crate tokenizers
?
Module
tokenizers
::
models
source
·
[
−
]
Expand description
Popular tokenizer models.
Modules
§
bpe
Byte Pair Encoding
model.
unigram
Unigram
model.
wordlevel
wordpiece
WordPiece
model.
Enums
§
ModelWrapper
TrainerWrapper