Module chunking

Text chunking functionality for processing large documents.

This module provides comprehensive text chunking capabilities to handle documents that exceed the language model’s context window. It supports multiple chunking strategies and overlap management to ensure no information is lost during processing.

Structs

ChunkResult

Result from processing a single chunk

ChunkingConfig

Configuration for text chunking

ResultAggregator

Result aggregator for combining extractions from multiple chunks

TextChunk

A chunk of text with metadata

TextChunker

Text chunker for processing large documents

Enums

ChunkingStrategy

Different strategies for chunking text