bpe-openai-0.3.0
- bpe-openai 0.3.0
- Permalink
- Docs.rs crate page
- MIT
- Links
- Repository
- crates.io
- Source
- Owners
- hendrikvanantwerpen
- gorzell
- aneubeck
- Dependencies
- bpe ^0.2 normal
  
  either ^1.13 normal
  
  regex-automata ^0.4 normal
  
  rmp-serde ^1 normal
  
  unicode-normalization ^0.1 normal
  
  bpe ^0.2 dev
  
  tiktoken-rs ^0.6 dev
  
  base64 ^0.22 build
  
  bpe ^0.2 build
  
  flate2 ^1.0 build
  
  rmp-serde ^1 build
  
  serde ^1 build
- Versions
- 69.57% of the crate is documented
Platform
Feature flags

docs.rs

Rust

bpe_openai0.3.0

Crate Items

Structs
Traits
Functions

List of all items

Structs

Pretokenizer
Tokenizer
appendable_encoder::AppendableEncoder
byte_pair_encoding::BytePairEncoding
interval_encoding::IntervalEncoding
normalizer::NormalizedString
prependable_encoder::PrependableEncoder

Traits

normalizer::Normalizable

Functions

cl100k_base
o200k_base
voyage3_base