Trait ChunkTokenizer 

pub trait ChunkTokenizer {
    // Required methods
    fn count_tokens(&self, text: &str) -> usize;
    fn max_tokens(&self) -> usize;
}

Token counting for HybridChunker — docling’s BaseTokenizer.

Required Methods

fn count_tokens(&self, text: &str) -> usize

Number of tokens in text (no special tokens).

fn max_tokens(&self) -> usize

The chunk budget (docling’s max_tokens, e.g. 256 for MiniLM).

Dyn Compatibility

This trait is dyn compatible.

In older versions of Rust, dyn compatibility was called "object safety".

Implementors