Docs.rs
tokenizers-0.12.0
tokenizers 0.12.0
Docs.rs crate page
Apache-2.0
Links
Homepage
Documentation
Repository
crates.io
Source
Owners
n1t0
Narsil
ArthurZucker
Dependencies
aho-corasick ^0.7
normal
cached-path ^0.5
normal
clap ^2.33
normal
derive_builder ^0.9
normal
dirs ^3.0
normal
esaxx-rs ^0.1
normal
indicatif ^0.15
normal
itertools ^0.9
normal
lazy_static ^1.4
normal
log ^0.4
normal
macro_rules_attribute ^0.0.2
normal
onig ^6.0
normal
paste ^1.0.6
normal
rand ^0.7
normal
rayon ^1.3
normal
rayon-cond ^0.1
normal
regex ^1.3
normal
regex-syntax ^0.6
normal
reqwest ^0.11
normal
serde ^1.0
normal
serde_json ^1.0
normal
spm_precompiled ^0.1
normal
thiserror ^1.0.30
normal
unicode-normalization-alignments ^0.1
normal
unicode-segmentation ^1.6
normal
unicode_categories ^0.1
normal
assert_approx_eq ^1.1
dev
criterion ^0.3
dev
tempfile ^3.1
dev
Versions
49.71%
of the crate is documented
This release has been yanked, go to latest version
Platform
x86_64-unknown-linux-gnu
Feature flags
Rust
About docs.rs
Privacy policy
Rust website
The Book
Standard Library API Reference
Rust by Example
The Cargo Guide
Clippy Documentation
☰
Module decoders
Modules
Enums
?
Module
tokenizers
::
decoders
source
·
[
−
]
Re-exports
pub use super::pre_tokenizers::
byte_level
;
pub use super::pre_tokenizers::
metaspace
;
Modules
bpe
ctc
wordpiece
Enums
DecoderWrapper