Docs.rs
tokenizers-0.19.1
tokenizers 0.19.1
Permalink
Docs.rs crate page
Apache-2.0
Links
Homepage
Documentation
Repository
Crates.io
Source
Owners
n1t0
Narsil
ArthurZucker
Dependencies
aho-corasick ^1.1
normal
derive_builder ^0.20
normal
esaxx-rs ^0.1.10
normal
fancy-regex ^0.13
normal
optional
getrandom ^0.2.10
normal
hf-hub ^0.3.2
normal
optional
indicatif ^0.17
normal
optional
itertools ^0.12
normal
lazy_static ^1.4
normal
log ^0.4
normal
macro_rules_attribute ^0.2.0
normal
monostate ^0.1.12
normal
onig ^6.4
normal
optional
paste ^1.0.14
normal
rand ^0.8
normal
rayon ^1.10
normal
rayon-cond ^0.3
normal
regex ^1.10
normal
regex-syntax ^0.8
normal
serde ^1.0
normal
serde_json ^1.0
normal
spm_precompiled ^0.1
normal
thiserror ^1.0.49
normal
unicode-normalization-alignments ^0.1
normal
unicode-segmentation ^1.11
normal
unicode_categories ^0.1
normal
assert_approx_eq ^1.1
dev
criterion ^0.5
dev
tempfile ^3.10
dev
Versions
47.13%
of the crate is documented
Platform
x86_64-unknown-linux-gnu
Feature flags
Rust
About docs.rs
Privacy policy
Rust website
The Book
Standard Library API Reference
Rust by Example
The Cargo Guide
Clippy Documentation
Rustdoc
1.79.0-nightly
(1cec373f6 2024-04-16)
Settings
?
Settings
Rustdoc settings
Back
You need to enable JavaScript be able to update your settings.