Struct LateChunkingPooler 

pub struct LateChunkingPooler { /* private fields */ }

Late chunking pooler: pools token embeddings into chunk embeddings.

This is the core operation of late chunking. Given token-level embeddings from a full document, it pools the tokens within each chunk boundary to create contextualized chunk embeddings.

Implementations

impl LateChunkingPooler

pub fn new(dim: usize) -> Self

Create a new late chunking pooler.

Arguments

• dim - Embedding dimension (e.g., 384 for all-MiniLM-L6-v2)

pub fn pool( &self, token_embeddings: &[Vec<f32>], chunks: &[Slab], doc_len: usize, ) -> Vec<Vec<f32>>

Pool token embeddings into chunk embeddings.

Arguments

• token_embeddings - Token-level embeddings from full document. Shape: [n_tokens, dim]. Each token has “seen” the full document.
• chunks - Chunk boundaries from any chunker.
• doc_len - Total document length in bytes (for mapping).

Returns

Contextualized chunk embeddings. Each chunk embedding is the mean of its constituent token embeddings.

Panics

Panics if token embeddings have inconsistent dimensions.

pub fn pool_with_offsets( &self, token_embeddings: &[Vec<f32>], token_offsets: &[(usize, usize)], chunks: &[Slab], ) -> Vec<Vec<f32>>

Pool with exact token-to-character mappings.

Use this when you have exact token offsets from the tokenizer, rather than relying on linear approximation.

Arguments

• token_embeddings - Token-level embeddings [n_tokens, dim].
• token_offsets - Character offset for each token [(start, end), …].
• chunks - Chunk boundaries.

Trait Implementations

impl Clone for LateChunkingPooler

fn clone(&self) -> LateChunkingPooler

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for LateChunkingPooler

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.