Module token

Token counting functionality using tiktoken-rs

Structs

FileTokenCount

Token count for a single file

TokenCounter

Token counter with caching support

TotalTokenEstimate

Total token estimate for multiple files

Functions

remaining_tokens

Calculate remaining token budget

would_exceed_limit

Check if adding a file would exceed token limit