Docs.rs
rust_tokenizers-7.0.2
rust_tokenizers 7.0.2
Docs.rs crate page
Apache-2.0
Links
Repository
crates.io
Source
Owners
guillaume-be
Dependencies
csv ^1.1.6
normal
hashbrown ^0.12.0
normal
itertools ^0.10.3
normal
lazy_static ^1.4.0
normal
protobuf =2.27.1
normal
rayon ^1.5.1
normal
regex ^1.5.5
normal
serde ^1.0.136
normal
serde_json ^1.0.79
normal
thiserror ^1.0.30
normal
unicode-normalization ^0.1.19
normal
unicode-normalization-alignments ^0.1.12
normal
anyhow ^1.0.56
dev
dirs ^4.0.0
dev
reqwest ^0.11.10
dev
tempfile ^3.3.0
dev
protobuf-codegen-pure ^2.27.1
build
Versions
91.97%
of the crate is documented
Go to latest version
Platform
i686-pc-windows-msvc
i686-unknown-linux-gnu
x86_64-apple-darwin
x86_64-pc-windows-msvc
x86_64-unknown-linux-gnu
Feature flags
Rust
About docs.rs
Privacy policy
Rust website
The Book
Standard Library API Reference
Rust by Example
The Cargo Guide
Clippy Documentation
Redirecting to
../../../rust_tokenizers/tokenizer/struct.Gpt2Tokenizer.html
...