deepstrike-tokenizer-0.2.71
- deepstrike-tokenizer 0.2.71
- Permalink
- Docs.rs crate page
- 20 September 2026
- Links
- crates.io
- Source
- Owners
- kongusen
- Dependencies
- tiktoken-rs ^0.6 normal
  pretty_assertions ^1 dev
- Versions
- 90% of the crate is documented
Platform
- x86_64-unknown-linux-gnu
Feature flags

docs.rs

Rust

Skip to main content

count_tokens

deepstrike_tokenizer0.2.71

deepstrike_tokenizer

Function count_tokens

pub fn count_tokens(text: &str) -> u32

Expand description

Quick helper: count tokens with cl100k_base.