Module extractors 

Format extractors & format-aware chunkers (extractors-v1, EPIC 12.13).

The front-door that turns a non-code document/data file into clean LLM text plus structure-aware chunks. It complements super::ingestion (which decides whether to index) by deciding how to read a given format:

Format	Extractor	Chunking
JSON	json	per array element / object entry
CSV/TSV	csv	header-prefixed row groups
EML	eml	header summary + body paragraphs
HTML	super::web::html_to_text	paragraphs of rendered Markdown
PDF	super::web::pdf	paragraphs of extracted text
text	(verbatim)	paragraphs

The text-based formats also register as named Chunkers in the extension_registry so they are discoverable via /v1/capabilities and exercised by the conformance suite. Every extractor degrades gracefully — arbitrary input never panics and non-empty input always yields at least one non-empty chunk.

Modules

csv

CSV/TSV → record text + row-group chunks (EPIC 12.13).

eml

RFC-822/2822 email (.eml) → header summary + body text (EPIC 12.13).

json

JSON → clean text + structure-aware chunks (EPIC 12.13).

Structs

Extracted

The result of extracting one document: a stable kind tag, clean text, and structure-aware chunks.

Functions

extract

Extract clean text + chunks from raw bytes, dispatching on path’s extension. Binary formats (PDF) read from bytes; text formats decode UTF-8 lossily so malformed encodings still produce content.

is_binary_document

Whether path is a binary document format that must be read through extract from raw bytes because it is not valid UTF-8 text. Text and structured formats (json/csv/eml/html/markdown/…) index fine as raw UTF-8; only true binary documents — currently PDF — need byte-level extraction before they can enter the text index. Grows as binary extractors (DOCX, XLSX, …) are added. Single source of truth for the indexer’s read path.

paragraph_chunks

Split text into paragraph chunks on blank-line boundaries, trimming and dropping empties. The shared fallback chunker for prose-like formats.

register_into

Register the text-based format chunkers into reg. Called from ExtensionRegistry::with_builtins so the formats are first-class, discoverable, and conformance-checked.