Docs.rs
tokenizers-0.10.1
tokenizers 0.10.1
Docs.rs crate page
Apache-2.0
Links
Repository
crates.io
Source
Owners
n1t0
Narsil
ArthurZucker
Dependencies
clap ^2.33.0
normal
indicatif ^0.14.0
normal
lazy_static ^1.3.0
normal
rand ^0.7.2
normal
rayon ^1.2.0
normal
regex ^1.3.1
normal
regex-syntax ^0.6.12
normal
serde ^1.0
normal
serde_json ^1.0
normal
unicode-normalization-alignments ^0.1.12
normal
unicode_categories ^0.1.1
normal
criterion ^0.3.0
dev
tempfile ^3.1
dev
Versions
Go to latest version
Platform
i686-pc-windows-msvc
i686-unknown-linux-gnu
x86_64-apple-darwin
x86_64-pc-windows-msvc
x86_64-unknown-linux-gnu
Feature flags
Rust
About docs.rs
Privacy policy
Rust website
The Book
Standard Library API Reference
Rust by Example
The Cargo Guide
Clippy Documentation
☰
All crates
1
2
3
pub
mod
iter
;
pub
mod
padding
;
pub
mod
truncation
;