Trait TokenizerBackend 

pub trait TokenizerBackend: Plugin {
    // Required method
    fn count_tokens(&self, text: &str) -> usize;
}

Trait for in-process tokenizer backend plugins.

Unlike crate::plugins::EmbeddingBackend, this trait is synchronous: the chunk splitter calls Self::count_tokens inside its boundary search, many times per chunk, so counting must be a direct call with no async dispatch. Host-language bridges (PyO3, napi-rs, etc.) invoke their host callable synchronously on the calling thread; implementations should keep count_tokens cheap — it dominates chunking time when the backend is slow.

Lifecycle

initialize() is called once during registration, before any count_tokens call; lazy-loading implementations should load their vocabulary there. After registration succeeds, count_tokens may be called from any thread, concurrently. shutdown() runs on unregistration and may overlap an in-flight count_tokens call from a chunking run that resolved the backend earlier — implementations must tolerate this, e.g. by keeping the resources count_tokens needs alive via Arc.

Thread safety

Backends must be Send + Sync + 'static (inherited from Plugin). They are stored in Arc<dyn TokenizerBackend> and called concurrently from xberg’s chunking pipeline. If the underlying tokenizer isn’t thread-safe, the backend must serialize access internally.

Contract

• count_tokens must return a non-zero count for non-empty text. The registry probes this once at registration and rejects backends that report zero — a zero count would make every span appear to fit any budget. At runtime, a zero count for non-empty text is not trusted: the chunker substitutes the character count and logs the substitution. (An implementation may still return 0 for the empty string.)
• count_tokens must not panic; return a best-effort count for text the tokenizer can’t fully process.
• Counting should be deterministic for a given input — the splitter may evaluate overlapping spans of the same text repeatedly.

Required Methods

fn count_tokens(&self, text: &str) -> usize

Count the tokens in text according to this backend’s tokenizer.

Dyn Compatibility

This trait is dyn compatible.

In older versions of Rust, dyn compatibility was called "object safety".

Implementors