Struct TokenCountSplitter 

pub struct TokenCountSplitter<C: TokenCounter + ?Sized + 'static = dyn TokenCounter> { /* private fields */ }

Recursive token-budget splitter.

Construct via Self::new with any Arc<C> where C: TokenCounter + ?Sized + 'static. Both concrete (Arc<TiktokenCounter>) and type-erased (Arc<dyn TokenCounter>) inputs are accepted — the splitter monomorphises per concrete counter for inlined hot-path dispatch, or falls through to dyn dispatch when the operator passes an erased Arc. Chain Self::with_chunk_size / Self::with_chunk_overlap / Self::with_separators for tuning. Cloning is cheap — the counter sits behind an Arc and the separator list is held by Arc<[String]>.

Implementations

impl<C: TokenCounter + ?Sized + 'static> TokenCountSplitter<C>

pub fn new(counter: Arc<C>) -> Self

Build with the supplied TokenCounter and the default 512-token / 64-token shape.

pub const fn with_chunk_size(self, chunk_size: usize) -> Self

Override the target chunk size in tokens.

pub const fn with_chunk_overlap(self, chunk_overlap: usize) -> Self

Override the overlap (in tokens) between consecutive chunks. Values at or above the chunk size silently clamp to chunk_size - 1 at split time so the recursion terminates.

pub fn with_separators<I, S>(self, separators: I) -> Self

where I: IntoIterator<Item = S>, S: Into<String>,

Override the separator priority list. Defaults to ["\n\n", "\n", " ", ""] — paragraph → line → word → unit fallback. Pipelines splitting source code or LaTeX ship alternative priorities.

pub const fn chunk_size(&self) -> usize

Effective chunk size in tokens.

pub const fn chunk_overlap(&self) -> usize

Effective chunk overlap in tokens.

pub const fn counter(&self) -> &Arc<C>

Borrow the wired token counter — surfaces TokenCounter::encoding_name for OTel attribute emission and operator diagnostics.

Trait Implementations

impl<C: Clone + TokenCounter + ?Sized + 'static> Clone for TokenCountSplitter<C>

fn clone(&self) -> TokenCountSplitter<C>

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl<C: TokenCounter + ?Sized + 'static> Debug for TokenCountSplitter<C>

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl<C: TokenCounter + ?Sized + 'static> TextSplitter for TokenCountSplitter<C>

fn name(&self) -> &'static str

Stable splitter identifier — surfaces on every produced chunk’s Lineage::splitter field. "recursive-character", "markdown-structure", "token-count", etc.

fn split(&self, document: &Document) -> Vec<Document>

Slice document and return the resulting chunks. Returning a single-element vec equal to the input is a valid no-op (e.g. when the document already fits the target size); an empty vec is also valid (e.g. content is whitespace-only).