rust_transformers-0.2.0
- rust_transformers 0.2.0
- Permalink
- Docs.rs crate page
- Apache-2.0
- Links
- Repository
- crates.io
- Source
- Owners
- guillaume-be
- Dependencies
- csv ^1.1 normal
  
  itertools ^0.8.2 normal
  
  lazy_static ^1.4.0 normal
  
  pyo3 ^0.8.3 normal
  
  rayon ^1.2.1 normal
  
  regex ^1.3.1 normal
  
  serde_json ^1.0.44 normal
  
  unicode-normalization ^0.1.8 normal
  
  tempfile ^3.1.0 dev
- Versions
- 0.81% of the crate is documented
Platform
Feature flags

docs.rs
- About docs.rs
- Privacy policy

Rust

Module openai_gpt_tokenizer

rust_transformers0.2.0

Module openai_gpt_tokenizer

Module Items

Structs

In rust_transformers::preprocessing::tokenizer

Modules

base_tokenizer
bert_tokenizer
ctrl_tokenizer
gpt2_tokenizer
openai_gpt_tokenizer
roberta_tokenizer

rust_transformers::preprocessing::tokenizer

Module openai_gpt_tokenizer

Settings

Help

Structs§

OpenAiGptTokenizer