aft-tokenizer-0.42.0
- aft-tokenizer 0.42.0
- Permalink
- Docs.rs crate page
- MIT
- Links
- crates.io
- Source
- Owners
- ualtinok
- Dependencies
- fancy-regex ^0.18 normal
  criterion ^0.8 dev
  serde ^1 dev
  serde_json ^1 dev
  regex ^1 build
- Versions
- 100% of the crate is documented
Platform
- x86_64-unknown-linux-gnu
Feature flags

docs.rs

Rust

Skip to main content

encode

aft_tokenizer0.42.0

Function encode

pub fn encode(text: &str) -> Vec<u32>

Expand description

Encode text into Claude token IDs.