Expand description
The block model as JSON: every block with its kind, page, box, text, spans and reading-order index, for a caller that chunks the document itself (#572). The block model as JSON, for callers that chunk it themselves (#572).
crate::markdown next door writes the document a RAG pipeline READS. This
writes the document a pipeline TAKES APART: every block with the kind it
was classified as, the page and the box it was drawn in, its text, the
spans it is made of and its place in the reading order. A caller that
chunks by heading, cites a paragraph by page, or draws a highlight over
what it retrieved needs the geometry, and markdown has none.
WRITTEN BY HAND AND NOT BY A DERIVE. serde is an optional dependency of
this crate and off by default, deliberately – the core engine stays
serialization-agnostic and the bindings that emit JSON turn it on. A
serializer that only existed under a feature flag would be a serializer
most callers of the library do not have, so this one writes its own bytes
and is always there.
WHAT MAKES IT DETERMINISTIC, which is the whole of what a golden test over
it rests on: the keys are written in a fixed order rather than iterated out
of a map, and every number goes through [number], which has one rule for
every value a f64 can hold.
A NON-FINITE NUMBER IS null. JSON has no NaN and no infinity, and the two
ways out of that are both worse than saying nothing: printing NaN makes a
file no parser accepts, and printing 0 states a coordinate the document
does not have. Block::bbox is already finite or [0, 0, 0, 0] by the
time it leaves crate::blocks, so this is about a span from a pathological
transform and about a block a caller built by hand.
THE KIND’S OWN FIELDS ARE FLAT, next to kind rather than nested under it:
level on a heading, row and column on a cell. Each object has exactly
one kind, so reading level after reading kind is unambiguous – and it
is one step instead of two for every filter a caller writes.
AND THERE IS NO FALLBACK ARM FOR A KIND THIS DOES NOT KNOW. BlockKind is
#[non_exhaustive] for the callers outside this crate, and inside it the
compiler still counts the variants – so a kind added later stops the build
here instead of leaving the block to a _ arm that would write it out as
something it is not. The name of a kind is a wire contract, and the day one
is added is the day somebody has to choose its name.
WHY THE SPANS ARE THE SIX NUMBERS AND NOT THE WHOLE TextSpan. The per-
glyph boxes on a span outweigh everything else in this file by an order of
magnitude – one box of four numbers per character – and nothing that
chunks a document reads them. The caller that does want them asks
extract_text_blocks, whose TextSpanInfo is the canonical wire form of a
span and carries all of it. This carries what a chunk needs to be placed:
where the run is, how big it is, and what it was set in.
Functions§
- to_json
- Every block as JSON: kind, page, box, text, spans and reading-order index.