Crate tokenx 

Vocab-free heuristic LLM token-count estimation.

tokenx estimates how many tokens a language model would read in a string. It uses no vocabulary or BPE table. It splits text into segments (words, whitespace runs, punctuation runs), classifies each segment, and assigns a token count per segment with a small set of chars-per-token rules. The result is an estimate, not an exact count. It is not measured against any specific tokenizer and can diverge from one on some text. See the limitations below.

Four functions make up the API:

• estimate_token_count returns the token estimate for a string.
• is_within_token_limit checks the estimate against a limit.
• slice_by_tokens extracts a token range, like Array.slice.
• split_by_tokens chunks text by a token budget, with optional overlap.

Examples

use tokenx::estimate_token_count;

assert_eq!(estimate_token_count("Hello, world! This is a short sentence."), 11);

Counting operates on UTF-16 code units for lengths and slicing, matching JavaScript string semantics. The CJK rule counts Unicode code points. For text in the Basic Multilingual Plane these agree.

Limitations

The count is a heuristic with no measured error bound, so leave headroom when checking a hard context limit. Two biases are worth knowing:

• A whole-segment number counts as 1 token whatever its length. Real tokenizers split long digit runs, so numeric-heavy text such as logs, tables, and JSON undercounts. A 15-digit number is 1 token here but several in a typical tokenizer.
• Japanese hiragana uses the default fallback rate while katakana and kanji count one token per code point, so hiragana-heavy text undercounts.

Structs

LanguageConfig

A language rule. When pattern matches a segment, the segment uses average_chars_per_token instead of the default.

SplitByTokensOptions

Options for crate::split_by_tokens. Adds chunk overlap.

TokenEstimationOptions

Options for token estimation.

Functions

approximate_token_sizeDeprecated

Returns the token estimate for text. Deprecated alias of estimate_token_count.

estimate_token_count

Estimates the number of tokens in text.

estimate_token_count_with

Estimates the number of tokens in text with custom options.

is_within_token_limit

Reports whether the estimate for text is at or below token_limit.

is_within_token_limit_with

Reports whether the estimate for text is at or below token_limit, with custom options.

slice_by_tokens

Extracts the substring covering token positions [start, end).

slice_by_tokens_with

Extracts a token range with custom options. See slice_by_tokens.

split_by_tokens

Splits text into chunks, each holding up to tokens_per_chunk tokens.

split_by_tokens_with

Splits text into chunks with custom options, including overlap. See split_by_tokens.