tiktoken-rs-0.12.0
- tiktoken-rs 0.12.0
- Permalink
- Docs.rs crate page
- MIT
- 12 August 2026
- Links
- Homepage
- Repository
- crates.io
- Source
- Owners
- zurawiki
- Dependencies
- anyhow ^1.0.102 normal
  async-openai ^0.34.0 normal optional
  base64 ^0.22.1 normal
  bstr ^1.12.1 normal
  dhat ^0.3.2 normal optional
  fancy-regex ^0.17.0 normal
  lazy_static ^1.5.0 normal
  regex ^1.12.3 normal
  rustc-hash ^2 normal
- Versions
- 62.07% of the crate is documented
Platform
- x86_64-unknown-linux-gnu
Feature flags

docs.rs

Rust

Skip to main content

byte_pair_split

tiktoken_rs0.12.0

Function byte_pair_split

pub fn byte_pair_split<'a>(
    piece: &'a [u8],
    ranks: &FxHashMap<Vec<u8>, Rank>,
) -> Vec<&'a [u8]>