rust_tokenizers-3.1.4
- rust_tokenizers 3.1.4
- Docs.rs crate page
- Apache-2.0
- Links
- Repository
- crates.io
- Source
- Owners
- guillaume-be
- Dependencies
- csv ^1.1 normal
  hashbrown ^0.7.2 normal
  itertools ^0.8.2 normal
  lazy_static ^1.4.0 normal
  protobuf =2.14.0 normal
  rayon ^1.2.1 normal
  regex ^1.3.1 normal
  serde ^1.0.106 normal
  serde_json ^1.0.44 normal
  unicode-normalization ^0.1.8 normal
  unicode-normalization-alignments ^0.1.12 normal
  dirs ^2.0.0 dev
  reqwest ^0.10.4 dev
  tempfile ^3.1.0 dev
  protobuf-codegen-pure ^2.8.1 build
- Versions
Go to latest version
Platform
Feature flags

Rust

☰

Module vocab

Modules

rust_tokenizers::preprocessing

[−][src]Module rust_tokenizers::preprocessing::vocab

Modules

albert_vocab
base_vocab
bert_vocab
bpe_vocab
gpt2_vocab
marian_vocab
openai_gpt_vocab
roberta_vocab
sentence_piece_vocab
sentencepiece_proto