Docs.rs
rust_tokenizers-4.0.0
rust_tokenizers 4.0.0
Docs.rs crate page
Apache-2.0
Links
Repository
crates.io
Source
Owners
guillaume-be
Dependencies
csv ^1.1
normal
hashbrown ^0.7.2
normal
itertools ^0.8.2
normal
lazy_static ^1.4.0
normal
protobuf =2.14.0
normal
rayon ^1.2.1
normal
regex ^1.3.1
normal
serde ^1.0.106
normal
serde_json ^1.0.44
normal
thiserror ^1.0.20
normal
unicode-normalization ^0.1.8
normal
unicode-normalization-alignments ^0.1.12
normal
anyhow ^1.0.32
dev
dirs ^2.0.0
dev
reqwest ^0.10.4
dev
tempfile ^3.1.0
dev
protobuf-codegen-pure ^2.8.1
build
Versions
10.41%
of the crate is documented
Go to latest version
Platform
i686-pc-windows-msvc
i686-unknown-linux-gnu
x86_64-apple-darwin
x86_64-pc-windows-msvc
x86_64-unknown-linux-gnu
Feature flags
Rust
About docs.rs
Privacy policy
Rust website
The Book
Standard Library API Reference
Rust by Example
The Cargo Guide
Clippy Documentation
☰
Module bert_tokenizer
Structs
rust_tokenizers
::
preprocessing
::
tokenizer
All crates
?
[
−
]
[src]
Module
rust_tokenizers
::
preprocessing
::
tokenizer
::
bert_tokenizer
Structs
BertTokenizer