Skip to main content

Crate text_chipper

Crate text_chipper 

Source
Expand description

text_chipper: token-budgeted chunking for Markdown and plain text.

The crate takes a &str and returns chunks that fit a token budget, never cut inside a grapheme cluster, and map back to the original input even when Unicode normalization is applied. Python bindings (feature python) are exposed through TextChipper.

§Scope

  • Input: Markdown and plain text only. There is no HTML, PDF or DOCX loader and no code AST (function/class boundaries). Convert such inputs to Markdown or text first. The file helpers (read_text_file, FileChunker) read UTF-8 text files only.

§Budget guarantee

  • strict = true (default): every chunk is re-tokenized exactly as it will be delivered (context prefix included) and token_count is exact. If a single grapheme cluster alone exceeds the budget, chunking fails with ChipError::ChunkTooLarge.
  • strict = false: that final exact check is skipped. This is faster, but there is no guarantee that every chunk fits, and token_count may be an estimate. An unsplittable grapheme cluster larger than the budget is emitted as its own chunk instead of an error.

§Two chunking modes (include_context)

  • include_context = true (default): the Markdown block parser runs. Headings are tracked and attached to chunks as headers; fenced code blocks and list items are kept intact when they fit; adjacent blocks are packed together (merge_blocks). Blocks are the blank-line-separated pieces of text: tables, block quotes, HTML blocks and indented code get no special handling. BlockParser is a chunking aid, not a complete CommonMark parser.
  • include_context = false (“flat mode”): smart_chunk and smart_chunk_mapped do not parse Markdown at all. The normalized text is cut into token windows that prefer paragraph, line and word boundaries; fences, lists and tables are not protected and headers is always empty. FileChunker gives the same chunks as smart_chunk on the file’s text, at any size; flat-mode files above 32 MiB are rejected with an error (stream them with include_context = true). See ChunkConfig.

Structs§

Block
A source-ordered Markdown block emitted by crate::BlockParser.
BlockParser
Incremental, line-oriented block splitter used for chunking. It is not a CommonMark parser. It splits text at blank lines, recognizes fenced code, ATX/short setext headings and YAML/TOML front matter, and keeps exact source offsets with bounded memory. Block quotes, HTML blocks, indented code, tables and link reference definitions are ordinary text blocks, and lists, quotes and thematic breaks do not interrupt a paragraph. List structure inside a block is found by crate::parse_list_items. Agreement with pulldown-cmark on the CommonMark corpus is measured by oracle_block_tests (see the README).
CharCursor
Chunk
A chunk of the normalized input.
ChunkConfig
Chunking options.
FileChunker
Chunks a UTF-8 text file. Iterate with FileChunker::next_chunk.
ListItem
Source range and nesting metadata for one list item.
NormBase
Running totals (normalized/original bytes and chars) before the text being normalized.
NormMap
Piecewise-linear map. Only segments that normalization actually changed are stored; between them offsets differ by a constant. A position strictly inside a changed segment (e.g. between the two letters of an NFKC-expanded ligature) maps to the segment start (chunk start) or the segment end (chunk end), so the original span always covers the whole chunk.
OrigSpan
Position of a chunk in the ORIGINAL (un-normalized) text.
Origin
SmartChunker
Incremental, block-aware chunker: feed it lines with SmartChunker::push_line, then call SmartChunker::finish. It drives the Markdown BlockParser, packs blocks into chunks and splits oversized blocks.

Enums§

Backend
BlockKind
The Markdown construct indicated by a block’s first line.
ChipError
Every failure is reported through this type; the Python bindings raise it as an exception (ValueError, RuntimeError or OSError) and never return partial or empty results.
ListMarker
The marker used to start a list item.
Normalization

Constants§

FLAT_FILE_MAX_BYTES
Largest file accepted with include_context = false (flat mode).

Traits§

Tokenizer
A user-supplied token counter. Implement this to plug any tokenizer (a model-specific SentencePiece/BPE, a remote counting service, a heuristic) into the chunker.

Functions§

context_prefix
The text prepended to a chunk’s body to form its contextualized text, e.g. [ctx:Guide > Install] . The prefix is counted against the token budget.
parse_list_items
Parse list item boundaries and nesting from Markdown source.
read_text_file
Read a complete file into one String. This convenience helper is intentionally whole-file and is not the bounded-memory API; use FileChunker for streaming.
smart_chunk
Chunk text; offsets in the result refer to the normalized text. See smart_chunk_mapped for original-text spans and ChunkConfig for semantics.
smart_chunk_mapped
Chunk text and also return, for every chunk, its span in the original text (before normalization). See ChunkConfig for the two chunking modes and the budget guarantee. Errors are returned, never swallowed.
split_by_tokens

Type Aliases§

Headers
Shared heading context attached to a parsed block.