claude-tokenizer-0.3.0
- claude-tokenizer 0.3.0
- Permalink
- Docs.rs crate page
- MIT
- Links
- Repository
- crates.io
- Source
- Owners
- Jellyfishboy
- Dependencies
- anyhow ^1.0.89 normal
  tokenizers ^0.20.0 normal
- Versions
- 0% of the crate is documented
Platform
- x86_64-unknown-linux-gnu
Feature flags

docs.rs

Rust

tokenize

claude_tokenizer0.3.0

claude_tokenizer

Function tokenize

pub fn tokenize(text: &str) -> Result<Vec<(u32, String)>>