tokenizers-0.22.2
- tokenizers 0.22.2
- Permalink
- Docs.rs crate page
- Apache-2.0
- Links
- Homepage
- Repository
- crates.io
- Source
- Owners
- danieldk
- n1t0
- Narsil
- ArthurZucker
- Dependencies
- ahash ^0.8.11 normal
  aho-corasick ^1.1 normal
  compact_str ^0.9 normal
  dary_heap ^0.3.6 normal
  derive_builder ^0.20 normal
  esaxx-rs ^0.1.10 normal
  fancy-regex ^0.14 normal optional
  getrandom ^0.3 normal
  hf-hub ^0.4.1 normal optional
  indicatif ^0.18 normal optional
  itertools ^0.14 normal
  log ^0.4 normal
  macro_rules_attribute ^0.2.0 normal
  monostate ^0.1.12 normal
  onig ^6.5.1 normal optional
  paste ^1.0.14 normal
  rand ^0.9 normal
  rayon ^1.10 normal
  rayon-cond ^0.4 normal
  regex ^1.10 normal
  regex-syntax ^0.8 normal
  serde ^1.0 normal
  serde_json ^1.0 normal
  spm_precompiled ^0.1.3 normal
  thiserror ^2 normal
  unicode-normalization-alignments ^0.1 normal
  unicode-segmentation ^1.11 normal
  unicode_categories ^0.1 normal
  assert_approx_eq ^1.1 dev
  criterion ^0.6 dev
  tempfile ^3.10 dev
  tracing ^0.1 dev
  tracing-subscriber ^0.3.18 dev
- Versions
- 45.41% of the crate is documented
Platform
- x86_64-unknown-linux-gnu
Feature flags

docs.rs

Rust

Module bert

tokenizers0.22.2

Module bert

Module Items

Structs

In tokenizers::normalizers

tokenizers::normalizers

Module bert

Structs§

BertNormalizer