claude-tokenizer-0.3.0
- claude-tokenizer 0.3.0
- Docs.rs crate page
- MIT
- Links
- Repository
- crates.io
- Source
- Owners
- Jellyfishboy
- Dependencies
- anyhow ^1.0.89 normal
  tokenizers ^0.20.0 normal
- Versions
- 0% of the crate is documented
Platform
- x86_64-unknown-linux-gnu
Feature flags

docs.rs

Rust

Crate claude_tokenizer

claude_tokenizer0.3.0

All Items

Crate Items

Functions

Crate claude_tokenizer

Functions§

count_tokens
get_tokenizer
tokenize