Enum TokenCounterKind 

pub enum TokenCounterKind {
    HuggingFace {
        model_id: String,
    },
    HuggingFaceFile {
        path: PathBuf,
    },
    TikToken,
    Word,
}

Selects which token counting implementation to use.

from_env() picks the best available counter based on env vars and the current embedding provider setting. WordCounter is the last-resort fallback.

Variants

HuggingFace

Accurate BPE/WordPiece via a HuggingFace tokenizer model ID (requires network or cache).

Fields

model_id: String

HuggingFaceFile

Accurate BPE/WordPiece from a local tokenizer.json file.

Fields

path: PathBuf

TikToken

TikToken cl100k_base BPE (for OpenAI models).

Word

Whitespace word count. Last-resort fallback.

Implementations

impl TokenCounterKind

pub fn from_env() -> Self

Determine the best available token counter from the environment.

Mirrors Python’s LiteLLMEmbeddingEngine.get_tokenizer() logic, which selects a tokenizer based on the provider and stores it on the engine instance. Python’s chunk_by_sentence() calls embedding_engine.tokenizer.count_tokens() directly — the tokenizer is a property of the engine, not a separate config. The Rust design decouples them (TokenCounterKind is independent of the engine), but the selection logic below preserves the same provider → tokenizer mapping.

Priority order (highest wins):

1. COGNEE_TOKEN_COUNTER=tiktoken → TikToken
2. COGNEE_TOKEN_COUNTER=huggingface or COGNEE_TOKEN_COUNTER=hf → check HUGGINGFACE_TOKENIZER
3. HUGGINGFACE_TOKENIZER env var is set → HuggingFace { model_id }
4. EMBEDDING_PROVIDER=onnx or fastembed and EMBEDDING_TOKENIZER_PATH is set and the file exists → HuggingFaceFile
5. EMBEDDING_PROVIDER=openai or openai_compatible → TikToken
6. EMBEDDING_PROVIDER=ollama and HUGGINGFACE_TOKENIZER set → HuggingFace
7. Fallback → Word

pub fn build(self) -> Result<Box<dyn TokenCounter + Send + Sync>, ChunkingError>

Construct a boxed TokenCounter from this kind.

Returns an error if the selected kind cannot be constructed (e.g. file not found, model download failed). When the relevant Cargo feature is disabled, silently falls back to WordCounter and logs a warning — so the crate compiles without optional features but users get a visible signal that their configured tokenizer is inactive.

Trait Implementations

impl Clone for TokenCounterKind

fn clone(&self) -> TokenCounterKind

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for TokenCounterKind

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl<'de> Deserialize<'de> for TokenCounterKind

fn deserialize<__D>(__deserializer: __D) -> Result<Self, __D::Error>

where __D: Deserializer<'de>,

Deserialize this value from the given Serde deserializer.

impl Serialize for TokenCounterKind

fn serialize<__S>(&self, __serializer: __S) -> Result<__S::Ok, __S::Error>

where __S: Serializer,

Serialize this value into the given Serde serializer.