Struct CountTokensCache 

pub struct CountTokensCache { /* private fields */ }

Available on crate features async and pricing only.

Bounded cache for count_tokens results, keyed by a stable hash of the request body. Use to skip the network round-trip on repeated previews against unchanged inputs (long-running agent sessions, IDE integrations, etc.).

Eviction is FIFO once capacity is reached – not a true LRU, but adequate for the common pattern of “preview the same prompt many times, occasionally see a new one.” Wrap in std::sync::Arc to share across tasks; the inner state is Mutex-protected.

Implementations

impl CountTokensCache

pub fn new(capacity: usize) -> Self

Build a new cache with the given capacity.

pub fn len(&self) -> usize

Number of entries currently stored.

pub fn is_empty(&self) -> bool

true when no entries are stored.

pub fn clear(&self)

Drop all cached entries.

pub fn get(&self, key: u64) -> Option<u32>

Look up a cached input-token count by request-hash.

pub fn put(&self, key: u64, value: u32)

Insert (or replace) an entry. Evicts the oldest entry by insertion order if the cache is at capacity and key is not already present.

Trait Implementations

impl Debug for CountTokensCache

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.