tokenizers-0.10.1
- tokenizers 0.10.1
- Docs.rs crate page
- Apache-2.0
- Links
- Repository
- crates.io
- Source
- Owners
- n1t0
- Narsil
- ArthurZucker
- Dependencies
- clap ^2.33.0 normal
  indicatif ^0.14.0 normal
  lazy_static ^1.3.0 normal
  rand ^0.7.2 normal
  rayon ^1.2.0 normal
  regex ^1.3.1 normal
  regex-syntax ^0.6.12 normal
  serde ^1.0 normal
  serde_json ^1.0 normal
  unicode-normalization-alignments ^0.1.12 normal
  unicode_categories ^0.1.1 normal
  criterion ^0.3.0 dev
  tempfile ^3.1 dev
- Versions
Go to latest version
Platform
Feature flags

Rust

☰

Module models

Modules

[−][src]Module tokenizers::models

Popular tokenizer models.

Modules

bpe	Byte Pair Encoding model.
wordlevel
wordpiece	WordPiece model.