Module extract 

Turning arbitrary document JSON into indexable plain text.

The extractor is a recursive string-leaf walk: it descends a serde_json::Value, concatenates every string it finds, skips object keys the rule excludes, and prefixes the values of keys the rule marks. It is deliberately typed on serde_json::Value and not on any app’s schema, so the same code serves RTDB documents today and Yjs/CRDT documents converted to JSON later.

Why a generic walk, and what it costs

notillo’s inline content is a union: string | [text, styleFlags] | [text, styleFlags, colors] | {l,c} | {wl,wt} | {tg}. The walk reproduces the app’s own extractBlockText() output, table cells included, but it also picks up style-flag codes ("b", "bi") as tokens, because nothing in the JSON distinguishes a styled-text tuple ["Hi","b"] from a table row ["Hi","there"].

The walk cannot infer that distinction, but a manifest can state it: a Selector::JsonPath filter picks exactly the nodes that carry prose ($.c[?@.t=='p'].text), and extract: "string" takes one verbatim instead of descending into its siblings. Where a manifest says nothing the walk stays the default and the trade stands — those flags are one- and two-character tokens that barely move bm25(), whereas a typed node-union DSL would have to guess at the same ambiguity and would silently truncate real table text when it guessed wrong. Genuinely harmful values — link targets, color codes — are removed by name, either by listing them in excludeKeys or, better, by listing the keys that do carry prose in keys: a denylist fails silently when the document schema grows a key nobody thought to add to it.

For the positional case itself — the style flag that no name can reach — the preferred lever is a part rule’s prune list, which deletes the tuple’s tail before this walk runs, so the walk stays one ordered rule and sees prose only. See crate::prune.

Structs

TextSink

Accumulates extracted text under a hard character budget.

Functions

extract_field

Extract the text a single FieldRule selects out of one document.

extract_fields

Extract every rule in rules into one sink, in declaration order.

resolve_path

Follow a pre-split dotted path. An empty path is the document itself.

resolve_str

Read a path as a plain scalar string — used for ids, parent links and sort keys, where a recursive walk would be wrong.