Docs.rs
tokenizers-0.23.1
tokenizers 0.23.1
Docs.rs crate page
Apache-2.0
Links
Homepage
Repository
crates.io
Source
Owners
danieldk
n1t0
Narsil
ArthurZucker
Dependencies
ahash ^0.8.11
normal
compact_str ^0.9
normal
daachorse ^1.0.1
normal
dary_heap ^0.3.6
normal
derive_builder ^0.20
normal
esaxx-rs ^0.1.10
normal
fancy-regex ^0.17
normal
optional
getrandom ^0.3
normal
hf-hub ^0.4.1
normal
optional
indicatif ^0.18
normal
optional
itertools ^0.14
normal
log ^0.4
normal
macro_rules_attribute ^0.2.0
normal
monostate ^0.1.12
normal
onig ^6.5.1
normal
optional
paste ^1.0.14
normal
rand ^0.9
normal
rayon ^1.10
normal
rayon-cond ^0.4
normal
regex ^1.10
normal
regex-syntax ^0.8
normal
serde ^1.0
normal
serde_json ^1.0
normal
spm_precompiled ^0.1.3
normal
thiserror ^2
normal
unicode-normalization-alignments ^0.1
normal
unicode-segmentation ^1.11
normal
unicode_categories ^0.1
normal
assert_approx_eq ^1.1
dev
criterion ^0.6
dev
tempfile ^3.10
dev
tracing ^0.1
dev
tracing-subscriber ^0.3.18
dev
Versions
45.83%
of the crate is documented
Platform
x86_64-unknown-linux-gnu
Feature flags
docs.rs
About docs.rs
Badges
Builds
Metadata
Shorthand URLs
Download
Rustdoc JSON
Build queue
Privacy policy
Rust
Rust website
The Book
Standard Library API Reference
Rust by Example
The Cargo Guide
Clippy Documentation
Skip to main content
Module pre_tokenizers
tokenizers
0.23.1
Module pre_
tokenizers
Module Items
Modules
Enums
In crate tokenizers
tokenizers
Module
pre_
tokenizers
Copy item path
Source
Modules
§
bert
byte_
level
delimiter
digits
fixed_
length
metaspace
punctuation
sequence
split
unicode_
scripts
whitespace
Enums
§
PreTokenizer
Wrapper