tokenizers-enfer-0.21.0-dev.0
- tokenizers-enfer 0.21.0-dev.0
- Docs.rs crate page
- Apache-2.0
- Links
- Homepage
- Repository
- crates.io
- Source
- Owners
- sssemil
- Dependencies
- aho-corasick ^1.1 normal
  derive_builder ^0.20 normal
  esaxx-rs ^0.1.10 normal
  fancy-regex ^0.13 normal optional
  getrandom ^0.2.10 normal
  hf-hub-enfer ^0.3.2 normal optional
  indicatif ^0.17 normal optional
  itertools ^0.12 normal
  lazy_static ^1.4 normal
  log ^0.4 normal
  macro_rules_attribute ^0.2.0 normal
  monostate ^0.1.12 normal
  onig ^6.4 normal optional
  paste ^1.0.14 normal
  rand ^0.8 normal
  rayon ^1.10 normal
  rayon-cond ^0.3 normal
  regex ^1.10 normal
  regex-syntax ^0.8 normal
  serde ^1.0 normal
  serde_json ^1.0 normal
  spm_precompiled ^0.1.3 normal
  thiserror ^1.0.49 normal
  unicode-normalization-alignments ^0.1 normal
  unicode-segmentation ^1.11 normal
  unicode_categories ^0.1 normal
  assert_approx_eq ^1.1 dev
  criterion ^0.5 dev
  tempfile ^3.10 dev
  tracing ^0.1 dev
  tracing-subscriber ^0.3.18 dev
- Versions
- 48.32% of the crate is documented
Go to latest stable release
Platform
- x86_64-unknown-linux-gnu
Feature flags

docs.rs

Rust

tokenizers0.21.0-dev.0

Module models

Module Items

Modules
Enums

In crate tokenizers

Module models

Expand description

Popular tokenizer models.

Modules§

bpe
Byte Pair Encoding model.
unigram
Unigram model.
wordlevel
wordpiece
WordPiece model.

Enums§

ModelWrapper
TrainerWrapper