mako-0.2.3
- mako 0.2.3
- Docs.rs crate page
- MIT OR Apache-2.0
- Links
- crates.io
- Source
- Owners
- jafioti
- Dependencies
- clap ^2.33 normal
  
  derive_builder ^0.9 normal
  
  esaxx-rs ^0.1 normal
  
  indicatif ^0.15 normal
  
  itertools ^0.9 normal
  
  lazy_static ^1.4 normal
  
  lentrait ^0.2.2 normal
  
  log ^0.4 normal
  
  multiqueue ^0.3.2 normal
  
  onig ^6.0 normal
  
  pyo3 ^0.13.2 normal
  
  rand ^0.8.3 normal
  
  rayon ^1.5.1 normal
  
  rayon-cond ^0.2.0 normal
  
  regex ^1.3 normal
  
  regex-syntax ^0.6 normal
  
  serde ^1.0 normal
  
  serde_json ^1.0.59 normal
  
  spm_precompiled ^0.1 normal
  
  thread-control ^0.1.2 normal
  
  unicode-normalization-alignments ^0.1 normal
  
  unicode-segmentation ^1.6 normal
  
  unicode_categories ^0.1 normal
  
  assert_approx_eq ^1.1 dev
  
  criterion ^0.3 dev
  
  tempfile ^3.1 dev
- Versions
- 81.13% of the crate is documented
Go to latest version
Platform
- x86_64-unknown-linux-gnu
Feature flags

Rust

☰

Other items in
mako::tokenization

Function mako::tokenization::batch_untokenize_bpe[−][src]

pub fn batch_untokenize_bpe(tokens: Vec<Vec<String>>) -> Vec<String>

Expand description

Untokenize BPE tokens