Docs.rs
tokenizers-enfer-0.21.1
tokenizers-enfer 0.21.1
Permalink
Docs.rs crate page
Apache-2.0
Links
Homepage
Repository
crates.io
Source
Owners
sssemil
Dependencies
aho-corasick ^1.1
normal
derive_builder ^0.20
normal
esaxx-rs ^0.1.10
normal
fancy-regex ^0.13
normal
optional
getrandom ^0.2.10
normal
hf-hub-enfer ^0.3.2
normal
optional
indicatif ^0.17
normal
optional
itertools ^0.12
normal
lazy_static ^1.4
normal
log ^0.4
normal
macro_rules_attribute ^0.2.0
normal
monostate ^0.1.12
normal
onig ^6.4
normal
optional
paste ^1.0.14
normal
rand ^0.8
normal
rayon ^1.10
normal
rayon-cond ^0.3
normal
regex ^1.10
normal
regex-syntax ^0.8
normal
serde ^1.0
normal
serde_json ^1.0
normal
spm_precompiled ^0.1.3
normal
thiserror ^1.0.49
normal
unicode-normalization-alignments ^0.1
normal
unicode-segmentation ^1.11
normal
unicode_categories ^0.1
normal
assert_approx_eq ^1.1
dev
criterion ^0.5
dev
tempfile ^3.10
dev
tracing ^0.1
dev
tracing-subscriber ^0.3.18
dev
Versions
48.17%
of the crate is documented
Platform
x86_64-unknown-linux-gnu
Feature flags
docs.rs
About docs.rs
Badges
Builds
Metadata
Shorthand URLs
Download
Rustdoc JSON
Build queue
Privacy policy
Rust
Rust website
The Book
Standard Library API Reference
Rust by Example
The Cargo Guide
Clippy Documentation
Module utils
tokenizers
0.21.1
Module utils
Module Items
Structs
In tokenizers::
normalizers
tokenizers
::
normalizers
Module
utils
Copy item path
Source
Structs
ยง
Lowercase
Lowercases the input
Sequence
Allows concatenating multiple other Normalizer as a Sequence. All the normalizers run in sequence in the given order against the same NormalizedString.