Docs.rs
tokenizers-0.15.2
tokenizers 0.15.2
Docs.rs crate page
Apache-2.0
Links
Homepage
Documentation
Repository
crates.io
Source
Owners
n1t0
Narsil
ArthurZucker
Dependencies
aho-corasick ^1.1
normal
clap ^4.4
normal
derive_builder ^0.12
normal
esaxx-rs ^0.1.10
normal
fancy-regex ^0.13
normal
getrandom ^0.2.10
normal
hf-hub ^0.3.2
normal
indicatif ^0.17
normal
itertools ^0.12
normal
lazy_static ^1.4
normal
log ^0.4
normal
macro_rules_attribute ^0.2.0
normal
monostate ^0.1.9
normal
onig ^6.4
normal
paste ^1.0.14
normal
rand ^0.8
normal
rayon ^1.8
normal
rayon-cond ^0.3
normal
regex ^1.9
normal
regex-syntax ^0.8
normal
serde ^1.0
normal
serde_json ^1.0
normal
spm_precompiled ^0.1
normal
thiserror ^1.0.49
normal
unicode-normalization-alignments ^0.1
normal
unicode-segmentation ^1.10
normal
unicode_categories ^0.1
normal
assert_approx_eq ^1.1
dev
criterion ^0.5
dev
tempfile ^3.8
dev
Versions
47.09%
of the crate is documented
Go to latest version
Platform
x86_64-unknown-linux-gnu
Feature flags
Rust
About docs.rs
Privacy policy
Rust website
The Book
Standard Library API Reference
Rust by Example
The Cargo Guide
Clippy Documentation
tokenizers
0.15.2
Module pre_tokenizers
Modules
Enums
In crate tokenizers
?
Module
tokenizers
::
pre_tokenizers
source
·
[
−
]
Modules
§
bert
byte_level
delimiter
digits
metaspace
punctuation
sequence
split
unicode_scripts
whitespace
Enums
§
PreTokenizerWrapper