Docs.rs
rust_tokenizers-3.1.6
rust_tokenizers 3.1.6
Docs.rs crate page
Apache-2.0
Links
Repository
crates.io
Source
Owners
guillaume-be
Dependencies
csv ^1.1
normal
hashbrown ^0.7.2
normal
itertools ^0.8.2
normal
lazy_static ^1.4.0
normal
protobuf =2.14.0
normal
rayon ^1.2.1
normal
regex ^1.3.1
normal
serde ^1.0.106
normal
serde_json ^1.0.44
normal
unicode-normalization ^0.1.8
normal
unicode-normalization-alignments ^0.1.12
normal
dirs ^2.0.0
dev
reqwest ^0.10.4
dev
tempfile ^3.1.0
dev
protobuf-codegen-pure ^2.8.1
build
Versions
Go to latest version
Platform
i686-pc-windows-msvc
i686-unknown-linux-gnu
x86_64-apple-darwin
x86_64-pc-windows-msvc
x86_64-unknown-linux-gnu
Feature flags
Rust
About docs.rs
Privacy policy
Rust website
The Book
Standard Library API Reference
Rust by Example
The Cargo Guide
Clippy Documentation
☰
Module gpt2_tokenizer
Structs
rust_tokenizers
::
preprocessing
::
tokenizer
All crates
[
−
]
[src]
Module
rust_tokenizers
::
preprocessing
::
tokenizer
::
gpt2_tokenizer
Structs
Gpt2Tokenizer