Expand description
text_chipper: token-budgeted chunking for Markdown and plain text.
The crate takes a &str and returns chunks that fit a token budget, never cut inside a
grapheme cluster, and map back to the original input even when Unicode normalization is
applied. Python bindings (feature python) are exposed through TextChipper.
§Scope
- Input: Markdown and plain text only. There is no HTML, PDF or DOCX loader and no
code AST (function/class boundaries). Convert such inputs to Markdown or text first.
The file helpers (
read_text_file,FileChunker) read UTF-8 text files only.
§Budget guarantee
strict = true(default): every chunk is re-tokenized exactly as it will be delivered (context prefix included) andtoken_countis exact. If a single grapheme cluster alone exceeds the budget, chunking fails withChipError::ChunkTooLarge.strict = false: that final exact check is skipped. This is faster, but there is no guarantee that every chunk fits, andtoken_countmay be an estimate. An unsplittable grapheme cluster larger than the budget is emitted as its own chunk instead of an error.
§Two chunking modes (include_context)
include_context = true(default): the Markdown block parser runs. Headings are tracked and attached to chunks asheaders; fenced code blocks and list items are kept intact when they fit; adjacent blocks are packed together (merge_blocks). Blocks are the blank-line-separated pieces of text: tables, block quotes, HTML blocks and indented code get no special handling.BlockParseris a chunking aid, not a complete CommonMark parser.include_context = false(“flat mode”):smart_chunkandsmart_chunk_mappeddo not parse Markdown at all. The normalized text is cut into token windows that prefer paragraph, line and word boundaries; fences, lists and tables are not protected andheadersis always empty.FileChunkergives the same chunks assmart_chunkon the file’s text, at any size; flat-mode files above 32 MiB are rejected with an error (stream them withinclude_context = true). SeeChunkConfig.
Structs§
- Block
- A source-ordered Markdown block emitted by
crate::BlockParser. - Block
Parser - Incremental, line-oriented block splitter used for chunking. It is not a CommonMark
parser. It splits text at blank lines, recognizes fenced code, ATX/short setext headings and
YAML/TOML front matter, and keeps exact source offsets with bounded memory. Block quotes,
HTML blocks, indented code, tables and link reference definitions are ordinary text blocks,
and lists, quotes and thematic breaks do not interrupt a paragraph. List structure inside a
block is found by
crate::parse_list_items. Agreement withpulldown-cmarkon the CommonMark corpus is measured byoracle_block_tests(see the README). - Char
Cursor - Chunk
- A chunk of the normalized input.
- Chunk
Config - Chunking options.
- File
Chunker - Chunks a UTF-8 text file. Iterate with
FileChunker::next_chunk. - List
Item - Source range and nesting metadata for one list item.
- Norm
Base - Running totals (normalized/original bytes and chars) before the text being normalized.
- NormMap
- Piecewise-linear map. Only segments that normalization actually changed are stored; between them offsets differ by a constant. A position strictly inside a changed segment (e.g. between the two letters of an NFKC-expanded ligature) maps to the segment start (chunk start) or the segment end (chunk end), so the original span always covers the whole chunk.
- Orig
Span - Position of a chunk in the ORIGINAL (un-normalized) text.
- Origin
- Smart
Chunker - Incremental, block-aware chunker: feed it lines with
SmartChunker::push_line, then callSmartChunker::finish. It drives the MarkdownBlockParser, packs blocks into chunks and splits oversized blocks.
Enums§
- Backend
- Block
Kind - The Markdown construct indicated by a block’s first line.
- Chip
Error - Every failure is reported through this type; the Python bindings raise it as an exception
(
ValueError,RuntimeErrororOSError) and never return partial or empty results. - List
Marker - The marker used to start a list item.
- Normalization
Constants§
- FLAT_
FILE_ MAX_ BYTES - Largest file accepted with
include_context = false(flat mode).
Traits§
- Tokenizer
- A user-supplied token counter. Implement this to plug any tokenizer (a model-specific SentencePiece/BPE, a remote counting service, a heuristic) into the chunker.
Functions§
- context_
prefix - The text prepended to a chunk’s body to form its contextualized text, e.g.
[ctx:Guide > Install]. The prefix is counted against the token budget. - parse_
list_ items - Parse list item boundaries and nesting from Markdown source.
- read_
text_ file - Read a complete file into one
String. This convenience helper is intentionally whole-file and is not the bounded-memory API; useFileChunkerfor streaming. - smart_
chunk - Chunk
text; offsets in the result refer to the normalized text. Seesmart_chunk_mappedfor original-text spans andChunkConfigfor semantics. - smart_
chunk_ mapped - Chunk
textand also return, for every chunk, its span in the original text (before normalization). SeeChunkConfigfor the two chunking modes and the budget guarantee. Errors are returned, never swallowed. - split_
by_ tokens
Type Aliases§
- Headers
- Shared heading context attached to a parsed block.