Expand description
purrdf-markdown — a Markdown document becomes an RDF 1.2 graph along
its own structure.
The whole law this crate applies is written out, clause by clause, in
the specification that ships beside it:
crates/markdown/SPEC.md.
What follows is that law in summary.
§Stand-off first, claims second
The slicer is a stand-off markup engine: the document’s bytes are
never touched, and everything the crate knows is a typed annotation
over a range of them. analyze returns that layer as a
Document — sections, units, citations, and the rows that lifted
nothing — and render projects it into claims. slice_markdown
is the two of them in a row, and remains the whole surface a caller
who only wants triples needs.
§A codec, not an index
The projection is the encode half of a codec: the emitted spans
cover every byte of the source. A unit carries its text over its
byte span; a section carries its own heading or movement line as a
typed verbatim literal over that line’s span; and every maximal run
of bytes neither covers — a blank run, a horizontal rule, a table
row, the newline a split cut landed on — is a structure node
carrying its bytes the same way. reconstruct is the decode half:
it writes each span’s bytes at its offset, requires the cover whole,
admits an overlap only where a continues edge declares it and the
shared bytes agree, and proves the result against the document
node’s own sourceDigest. A graph and its document are therefore
two spellings of the same bytes, and either proves the other.
Every verbatim literal is typed, never plain, so what a full-text or embedding index selecting plain strings sees stays exact: one text per unit, one heading’s words per section, and not a byte of structure. The heading’s words are plain because they are content — an index that never saw a chapter title would report absence for the densest text a document carries — while the marks, the whitespace and the newline of the heading line travel only in its typed verbatim literal.
A Document carries the Profile it was admitted under
(Document::profile), and render takes the document alone.
That is not a convenience: the projection’s infallibility rests on
every law it applies having been answered for on these bytes, and a
profile passed in separately would be a law nothing checked — an
anchor admitted as a literal because no canon base was declared
would mint an unchecked IRI under a base handed in later.
The order is the point. The model is the finding; the graph is one
view of it. A consumer that wants to ask which unit covers byte
4,821 (Document::unit_at), which units a byte range touches
(Document::covering), how two spans lie against each other
(span_relation, the thirteen Allen interval relations), what a
unit’s exact quote and surrounding context are (Unit::anchor),
or which concordance rows named verses this document does not carry
(Document::unmatched_citations) asks the model, and never parses
a triple to do it.
§The dialect
One document node; one section node per ATX heading (# through
######) and per movement marker (⁂ *name*, U+2042, which sits one
level under the nearest heading so consecutive movements are
siblings); and one unit node per numbered verse (12. text) or
blank-line paragraph. Horizontal rules and table rows are structure,
not units. A unit carries exactly one plain xsd:string literal, the
verbatim byte span of the source, plus its byte span, its scalar
span, its content digest, its ordinal, its section, and its heading
lineage as typed literals, so a text index sees one text per unit and
a reader can always get back to the exact bytes — and prove, from the
digest, that the bytes it got back are the bytes the node was minted
over.
Structure is recognized on a line’s trimmed text, so a document
written with CRLF endings slices into the structure its LF twin
slices into while every span still carries the document’s own bytes,
the \r among them; and a verse number is a u64, so a longer run
of digits is no verse number at all and the line stays a paragraph.
The leading edge is bounded rather than trimmed: a heading, a
movement marker and a verse number are read behind up to three
leading spaces, while four or more — or a tab anywhere in that run,
which CommonMark expands to column four — open no marker and leave
the line as ordinary content. A byte order mark opening the document
is a statement about the encoding, so the first line’s structure is
read after it while every span still counts the document’s own
bytes; anywhere else the mark is ordinary content. All three are
read for recognition alone: the \r, the leading spaces and the
mark are exactly where the source put them, in every span and in
every literal.
Everything in that paragraph is Markdown, and it is all in one
module — the crate’s dialect, internal and documented at length in
the source. The split, the identities, the containment lattice, and
the projection know nothing about a heading or a verse, and the next
structured format plugs in at that seam.
§The split
A unit over max_bytes splits into pieces of at most that many
bytes, with no exception. Each cut falls at the last newline at or
before the bound — the cut lands on that newline and no piece
carries it — else at the last scalar boundary at or before the
bound. A continuation reaches back overlap bytes, snapped backward
to a line start, else to a scalar boundary, and never before the
unit’s own start. A cut never falls inside a scalar and a split never
crosses a heading. max_bytes is at least MIN_MAX_BYTES — the
widest UTF-8 scalar — and overlap sits strictly under it.
§The concordance
A ## Concordance table (| Verses | Canon source | Anchors |)
lifts into citation triples from each verse in a range to its
anchors. Inside the concordance is containment, not innermost: a
concordance organised into subsections holds its rows at a depth, and
a row at any depth under the heading is a citation row, while a
table no concordance section holds is structure and lifts nothing. A
row’s cells are delimited by its unescaped pipes only — \| is a
literal | in a cell and \\ a literal \, and any other backslash
is content — so a path or a note may carry a pipe. Each row’s lift
onto one verse is also its own
content-addressed citation node, which rdf:reifies the RDF 1.2
triple term <<( <unit> <cites> <anchor> )>> and carries that row’s
source paths — so a verse two rows cover keeps each row’s paths
beside that row’s anchors, which a flat projection could not say.
Every citation node also states its class and a back-edge to the unit
it is an edge of, and states them whatever its row lifted: a row that
names sources and no anchors reifies nothing, and those two lines
are what keep its node — and so its sources — reachable from the unit
rather than emitted into a corner of the graph nothing points at. A
concordance may cover a canon wider than the document that carries
it: a row whose verses are all elsewhere lifts nothing here and is
not an error — it is reported as data, along with any row too
malformed to read at all.
§Identity
The output is a pure, deterministic function of the bytes and a
declared Profile: the same input slices to byte-identical claims
on every target, and every parameter of the law — the canon base
among them — is inside Profile::contract_id, so a change re-mints
it rather than drifting.
A unit’s IRI is <node base>unit:sha256:<hex>, the hex being the
SHA-256 of a length-prefixed preimage of the kind, the source id,
the profile’s chunking id, the byte span, the digest algorithm
tag, and the SHA-256 of the span’s bytes. Two byte-identical paragraphs in
one document are two nodes, because the span is inside the identity;
an insertion moves every later span and re-mints every later unit
while leaving their text untouched; a same-length substitution
re-mints only the unit it touches and moves no boundary. The digest
algorithm is inside both the preimage and the IRI, so another
producer can mint the same shape under another algorithm without
redefining the preimage.
A citation node is minted the same way over the pair it names — the concordance row’s own line span, and a digest of that line together with the unit’s IRI and the anchor terms the row mints — so a row that lifts onto two verses is two nodes, a unit that re-mints re-mints every citation of it, and a row read under two canon bases is two nodes rather than one node reifying two different triples.
§Two preimages, three ids, and which is which
One preimage cannot do both jobs, so there are two.
Profile::chunking_bytes states only what decides where a unit
starts and ends: the name, the version, the dialect grammar, and the
split law with its constants. Profile::emission_bytes is those
bytes verbatim followed by everything else that decides a byte of the
graph — the vocabulary, the canon base (its absence stated, not
implied), the concordance lift, the identity formulas and the
emission law.
From them, three ids, and no two of them are ever equal:
Profile::chunking_id, over the chunking preimage, is the third field of every node identity. A node is addressed by the law that cut it, so declaring a canon base or renaming a predicate leaves every unit and section where it was and the two graphs merge.Profile::contract_idis the law id, over the emission preimage, and it is what the graph states assliceProfile. Two runs that agree on it agree on every byte they emit — which is why the canon base is inside it: a declared base turns every citation object from a typed literal into a minted IRI.Profile::purremb_chunking_stagecarries the chunking preimage as a PURREMB stage, whose canonical form is TLV-framed and which an embedding family therefore can reproduce — the id derived from it is the one a chunk target carries. Neither of the other two is a chunking-stage id and no family can derive them; a consumer that wires one into a family has named an id nothing on the other side can check.
The split is what a consumer feels. Embeddings are addressed by the chunking id, so they survive a renamed term and a newly declared canon base and are invalidated exactly when a unit’s bytes move.
Unit::text_chunk_target hands a unit over as the kernel’s own
chunk subject, and verify_unit proves a unit against bytes by
delegating to the kernel’s reference verification law, so this crate
and a .purremb consumer refuse the same bytes for the same reason.
Document::byte_to_scalar and Document::scalar_to_byte convert
between the bytes the identities are in and the scalars a host
indexes text by.
§It mints no vocabulary
PurRDF is not an ontology. Every class, predicate, and datatype IRI
this crate emits, and the base under which it mints node IRIs, is
caller-supplied configuration in a Vocabulary; there is no
default namespace and no fabricated fallback. Vocabulary::under
derives a whole vocabulary from one base IRI with the crate’s local
names, and a caller who wants other IRIs sets the fields directly.
Vocabulary::standard derives it from the one namespace the
specification designates for documents meant to be exchanged — an
opinion about interchange, asked for by name, never reached for on a
caller’s behalf. The vocabulary is part of the profile’s identity:
the same law under two vocabularies is two profiles, and the local
names are inside it beside the base, so a renamed term is a law that
says it changed.
No derived IRI wears two roles. The classes, the predicates and the
datatypes are three disjoint sets — a heading’s text is carried by
heading and typed with headingText — because a namespace offered
for interchange has to be describable, and no OWL or SHACL document
can declare one IRI both an owl:DatatypeProperty and an
rdfs:Datatype. A caller setting the fields itself is sovereign over
its own terms and may spend fewer of them.
Caller-supplied is not unchecked. Every IRI a caller states — each
field of the vocabulary, the source id, the canon base, and every
IRI a concordance anchor mints under that base — must be an
absolute IRI under the workspace IRI law, which this crate
reaches through purrdf-core so that it is asking the same question
the kernel will ask when it interns the result. A relative IRI would
otherwise slice whole and be refused a stage later, far from the
field that wrote it.
That question has two halves and they are refused apart. A string
the law reads whole and finds scheme-less is relative, and the
remedy is to write it absolute. A string the law cannot read at all
is malformed, and the refusal carries the law’s own finding —
which byte, and what about it — because this crate does not own that
law and has no business rewording it. https://example.org/% is a
truncated percent-encoding, not an id “with no scheme”.
Re-exports§
pub use syntax::AuthoringDocument;pub use syntax::CodeSpanLimits;pub use syntax::SyntaxDocument;pub use syntax::SyntaxError;pub use syntax::SyntaxLimits;pub use syntax::replace_code_spans;pub use syntax::syntax;
Modules§
- syntax
- Bounded source syntax shared by authoring consumers and the RDF slicer.
Structs§
- Citation
- One concordance row, unflattened: the verse range it covers, its canon sources, its anchors, and what it lifted in this document.
- Claim
- One node’s triples, rendered as sorted N-Triples lines (valid
Turtle; declare
text/turtleto a consumer). - Content
Anchor - A unit’s text with bounded context on each side: the anchor a consumer re-finds the unit by after the document has been edited.
- Document
- A document as a typed layer of annotations over untouched source
bytes: the stand-off model
analyzereturns. - Malformed
Row - A row of a concordance table that could not be read as a citation at all, and why.
- Profile
- The slicing profile: its name and version, its vocabulary, the byte bound and overlap that act only on an oversize unit, and the optional canon IRI base for concordance anchors.
- Section
- A section of the document: a heading, or a movement marker.
- Source
Document - One source document: its IRI and its exact bytes.
- Span
- A byte range of the source:
startinclusive,endexclusive. - Unit
- A unit: a numbered verse, a paragraph, or one piece of either after the oversize split.
- Verbatim
Span - One text-carrying span of a cover: the bytes at
byte_start..byte_end, quoted verbatim. - Vocabulary
- Every IRI the slicer emits, supplied by the caller. The fields are
public so a caller can set any of them;
Vocabulary::underfills all of them from one base,Vocabulary::standardfills them from the namespace the specification designates, andVocabulary::validateis applied before any claim is rendered.
Enums§
- Claim
Kind - Which node a claim describes.
- Decode
Error - Why a graph could not be read back into a cover: the refusals of
the graph half of the decode law. The span half’s refusals
arrive wrapped, in
DecodeError::Reconstruct. - Markdown
Error - Why a document could not be sliced.
- Reconstruct
Error - Why a cover could not be decoded back into a document.
- RowDefect
- What a concordance row failed at.
- Span
Relation - The thirteen ways two byte spans can lie against each other: Allen’s interval relations, with the six inverses named as their own variants.
Constants§
- CONTEXT_
BYTES - How many bytes of context a unit’s content anchor reaches on each side, snapped to a scalar boundary so the context is always text.
- DEFAULT_
MAX_ BYTES - The default byte bound: a unit over this many bytes splits.
- DEFAULT_
OVERLAP - The default overlap: the continuation of a split reaches back this many bytes, snapped backward to a newline.
- DIGEST_
ALGORITHM - The digest algorithm tag carried inside every unit and section identity, in both the preimage and the IRI.
- MIN_
MAX_ BYTES - The least byte bound a profile may declare: the widest UTF-8 scalar is four bytes, and no piece is ever over the bound, so a smaller bound could not be honoured without cutting inside a scalar.
- PURREMB_
PARAMETER_ ENCODING - The parameter encoding
Profile::purremb_chunking_stagedeclares: the stage’s parameters are exactly the bytes ofProfile::chunking_bytes, which are UTF-8 plain text, one fact of the law per line. - RDF_
REIFIES rdf:reifies, the RDF 1.2 predicate binding a citation node to the triple term it reifies. LikeRDF_TYPEit is the standard’s IRI and never a vocabulary field: reification is a shape of the data model, not a term a deployment gets to rename.rdf:reifies— the RDF 1.2 predicate linking a reifier to the triple term it reifies.- RDF_
TYPE rdf:type, one of the four IRIs the slicer emits that are the standard’s, not the caller’s.rdf:type— the instance-of predicate.- STANDARD_
NAMESPACE - The namespace the specification designates for this crate’s terms:
the one namespace
Vocabulary::standardderives everything under. - XSD_
HEX_ BINARY xsd:hexBinary, the datatype of a unit’s content digest stated as data. The lexical form is the same lowercase hex the unit’s IRI carries, so a consumer compares the two without re-encoding either.xsd:hexBinary— a sequence of hex-encoded bytes.- XSD_
INTEGER xsd:integer, the datatype of every ordinal, level, byte offset, and scalar offset.xsd:integer— arbitrary-magnitude (this crate:i128-bounded) signed integer.
Functions§
- analyze
- Reads a Markdown document into the typed stand-off model under a profile: the seam every refusal is stated at.
- analyze_
syntax - Project an already admitted syntax model without parsing Markdown again. Source bytes come from that model, so ranges cannot be paired with another input.
- analyze_
with_ limits - Analyze through the shared native syntax scanner with explicit caller limits.
The source-syntax model can also be retained and projected with
analyze_syntax. - citation_
iri - The identity of one citation edge: what a concordance row lifted onto one unit.
- decode_
document - The document back from its graph: the specification’s two
extraction rules over a parsed dataset, then the kernel’s
reconstructover what they extracted. - reconstruct
- The document back from its cover: writes each span’s bytes at its
offset, requires every byte covered, admits an overlap only where a
continuesedge declares it and its bytes agree, and proves the result against the stated source digest. - render
- Projects an analyzed document into claims under the law it was
admitted under:
Document::profile, and no other. - section_
iri - The identity of a section, over its heading line’s bytes, under the
same chunking id
unit_iritakes. - slice_
markdown - Slices a Markdown document into claims under a profile:
analyzethenrender. - span_
relation - The Allen interval relation of
atob, over half-open byte spans. - structure_
iri - The identity of a structure node, over its own bytes, under the same
chunking id and the same seven-field preimage
unit_iritakes — only the kind differs. - unit_
iri - The identity of a unit, with the digest algorithm inside both the preimage and the IRI. A consumer re-derives it from the bytes at a span to prove a hit is bound to its source.
- verify_
unit - Proves a unit against the bytes it claims to annotate: the span is inside them, it falls on scalar boundaries at both ends, and the bytes at it still digest to the digest the unit carries.