Skip to main content

Module chunking

infiniloom_engine::document

Module chunking 

Source
Expand description

Document chunking for multi-turn LLM conversations.

Splits large documents into token-budgeted chunks that respect section boundaries and never split tables or lists.

Structs§

ChunkConfig
Configuration for document chunking.
DocumentChunk
A chunk of a document with metadata for multi-turn use.

Functions§

chunk_document
Split a document into token-budgeted chunks.
format_chunk_text
Format a single chunk as readable text with header.
format_chunks_json
Format chunks as a JSON array.