tokenizers-0.12.0
- tokenizers 0.12.0
- Docs.rs crate page
- Apache-2.0
- Links
- Homepage
- Documentation
- Repository
- crates.io
- Source
- Owners
- n1t0
- Narsil
- ArthurZucker
- Dependencies
- aho-corasick ^0.7 normal
  cached-path ^0.5 normal
  clap ^2.33 normal
  derive_builder ^0.9 normal
  dirs ^3.0 normal
  esaxx-rs ^0.1 normal
  indicatif ^0.15 normal
  itertools ^0.9 normal
  lazy_static ^1.4 normal
  log ^0.4 normal
  macro_rules_attribute ^0.0.2 normal
  onig ^6.0 normal
  paste ^1.0.6 normal
  rand ^0.7 normal
  rayon ^1.3 normal
  rayon-cond ^0.1 normal
  regex ^1.3 normal
  regex-syntax ^0.6 normal
  reqwest ^0.11 normal
  serde ^1.0 normal
  serde_json ^1.0 normal
  spm_precompiled ^0.1 normal
  thiserror ^1.0.30 normal
  unicode-normalization-alignments ^0.1 normal
  unicode-segmentation ^1.6 normal
  unicode_categories ^0.1 normal
  assert_approx_eq ^1.1 dev
  criterion ^0.3 dev
  tempfile ^3.1 dev
- Versions
- 49.71% of the crate is documented
This release has been yanked, go to latest version
Platform
- x86_64-unknown-linux-gnu
Feature flags

Rust

Module bpe

Structs

Module tokenizers::decoders::bpe

source · [−]

Structs

Allows decoding Original BPE by joining all the tokens and then replacing the suffix used to identify end-of-words by whitespaces