Module tokens 

Token estimation for LLM context budgeting.

Provides a zero-dependency Unicode-script-based heuristic for estimating token counts, useful for budget management without pulling in tiktoken.

use llm_kernel::tokens::estimate_tokens;

let count = estimate_tokens("Hello, world! こんにちは世界");
assert!(count > 0);

Re-exports

pub use chunk::ChunkOptions;

pub use chunk::chunk_text;

Modules

budget

Thread-safe token budget tracker. Token budget tracking for LLM context windows.

chunk

Document chunking by sentence boundary and token budget. Document chunking by sentence boundary and token budget.

Functions

estimate_tokens

Estimate the number of tokens in a string using Unicode-script heuristics.

estimate_tokens_min

Estimate tokens for a single string, returning at least min.