Skip to main content

Crate purrdf_markdown

Crate purrdf_markdown 

Source
Expand description

purrdf-markdown — a Markdown document becomes an RDF 1.2 graph along its own structure.

The whole law this crate applies is written out, clause by clause, in the specification that ships beside it: crates/markdown/SPEC.md. What follows is that law in summary.

§Stand-off first, claims second

The slicer is a stand-off markup engine: the document’s bytes are never touched, and everything the crate knows is a typed annotation over a range of them. analyze returns that layer as a Document — sections, units, citations, and the rows that lifted nothing — and render projects it into claims. slice_markdown is the two of them in a row, and remains the whole surface a caller who only wants triples needs.

§A codec, not an index

The projection is the encode half of a codec: the emitted spans cover every byte of the source. A unit carries its text over its byte span; a section carries its own heading or movement line as a typed verbatim literal over that line’s span; and every maximal run of bytes neither covers — a blank run, a horizontal rule, a table row, the newline a split cut landed on — is a structure node carrying its bytes the same way. reconstruct is the decode half: it writes each span’s bytes at its offset, requires the cover whole, admits an overlap only where a continues edge declares it and the shared bytes agree, and proves the result against the document node’s own sourceDigest. A graph and its document are therefore two spellings of the same bytes, and either proves the other.

Every verbatim literal is typed, never plain, so what a full-text or embedding index selecting plain strings sees stays exact: one text per unit, one heading’s words per section, and not a byte of structure. The heading’s words are plain because they are content — an index that never saw a chapter title would report absence for the densest text a document carries — while the marks, the whitespace and the newline of the heading line travel only in its typed verbatim literal.

A Document carries the Profile it was admitted under (Document::profile), and render takes the document alone. That is not a convenience: the projection’s infallibility rests on every law it applies having been answered for on these bytes, and a profile passed in separately would be a law nothing checked — an anchor admitted as a literal because no canon base was declared would mint an unchecked IRI under a base handed in later.

The order is the point. The model is the finding; the graph is one view of it. A consumer that wants to ask which unit covers byte 4,821 (Document::unit_at), which units a byte range touches (Document::covering), how two spans lie against each other (span_relation, the thirteen Allen interval relations), what a unit’s exact quote and surrounding context are (Unit::anchor), or which concordance rows named verses this document does not carry (Document::unmatched_citations) asks the model, and never parses a triple to do it.

§The dialect

One document node; one section node per ATX heading (# through ######) and per movement marker (⁂ *name*, U+2042, which sits one level under the nearest heading so consecutive movements are siblings); and one unit node per numbered verse (12. text) or blank-line paragraph. Horizontal rules and table rows are structure, not units. A unit carries exactly one plain xsd:string literal, the verbatim byte span of the source, plus its byte span, its scalar span, its content digest, its ordinal, its section, and its heading lineage as typed literals, so a text index sees one text per unit and a reader can always get back to the exact bytes — and prove, from the digest, that the bytes it got back are the bytes the node was minted over.

Structure is recognized on a line’s trimmed text, so a document written with CRLF endings slices into the structure its LF twin slices into while every span still carries the document’s own bytes, the \r among them; and a verse number is a u64, so a longer run of digits is no verse number at all and the line stays a paragraph. The leading edge is bounded rather than trimmed: a heading, a movement marker and a verse number are read behind up to three leading spaces, while four or more — or a tab anywhere in that run, which CommonMark expands to column four — open no marker and leave the line as ordinary content. A byte order mark opening the document is a statement about the encoding, so the first line’s structure is read after it while every span still counts the document’s own bytes; anywhere else the mark is ordinary content. All three are read for recognition alone: the \r, the leading spaces and the mark are exactly where the source put them, in every span and in every literal.

Everything in that paragraph is Markdown, and it is all in one module — the crate’s dialect, internal and documented at length in the source. The split, the identities, the containment lattice, and the projection know nothing about a heading or a verse, and the next structured format plugs in at that seam.

§The split

A unit over max_bytes splits into pieces of at most that many bytes, with no exception. Each cut falls at the last newline at or before the bound — the cut lands on that newline and no piece carries it — else at the last scalar boundary at or before the bound. A continuation reaches back overlap bytes, snapped backward to a line start, else to a scalar boundary, and never before the unit’s own start. A cut never falls inside a scalar and a split never crosses a heading. max_bytes is at least MIN_MAX_BYTES — the widest UTF-8 scalar — and overlap sits strictly under it.

§The concordance

A ## Concordance table (| Verses | Canon source | Anchors |) lifts into citation triples from each verse in a range to its anchors. Inside the concordance is containment, not innermost: a concordance organised into subsections holds its rows at a depth, and a row at any depth under the heading is a citation row, while a table no concordance section holds is structure and lifts nothing. A row’s cells are delimited by its unescaped pipes only — \| is a literal | in a cell and \\ a literal \, and any other backslash is content — so a path or a note may carry a pipe. Each row’s lift onto one verse is also its own content-addressed citation node, which rdf:reifies the RDF 1.2 triple term <<( <unit> <cites> <anchor> )>> and carries that row’s source paths — so a verse two rows cover keeps each row’s paths beside that row’s anchors, which a flat projection could not say. Every citation node also states its class and a back-edge to the unit it is an edge of, and states them whatever its row lifted: a row that names sources and no anchors reifies nothing, and those two lines are what keep its node — and so its sources — reachable from the unit rather than emitted into a corner of the graph nothing points at. A concordance may cover a canon wider than the document that carries it: a row whose verses are all elsewhere lifts nothing here and is not an error — it is reported as data, along with any row too malformed to read at all.

§Identity

The output is a pure, deterministic function of the bytes and a declared Profile: the same input slices to byte-identical claims on every target, and every parameter of the law — the canon base among them — is inside Profile::contract_id, so a change re-mints it rather than drifting.

A unit’s IRI is <node base>unit:sha256:<hex>, the hex being the SHA-256 of a length-prefixed preimage of the kind, the source id, the profile’s chunking id, the byte span, the digest algorithm tag, and the SHA-256 of the span’s bytes. Two byte-identical paragraphs in one document are two nodes, because the span is inside the identity; an insertion moves every later span and re-mints every later unit while leaving their text untouched; a same-length substitution re-mints only the unit it touches and moves no boundary. The digest algorithm is inside both the preimage and the IRI, so another producer can mint the same shape under another algorithm without redefining the preimage.

A citation node is minted the same way over the pair it names — the concordance row’s own line span, and a digest of that line together with the unit’s IRI and the anchor terms the row mints — so a row that lifts onto two verses is two nodes, a unit that re-mints re-mints every citation of it, and a row read under two canon bases is two nodes rather than one node reifying two different triples.

§Two preimages, three ids, and which is which

One preimage cannot do both jobs, so there are two. Profile::chunking_bytes states only what decides where a unit starts and ends: the name, the version, the dialect grammar, and the split law with its constants. Profile::emission_bytes is those bytes verbatim followed by everything else that decides a byte of the graph — the vocabulary, the canon base (its absence stated, not implied), the concordance lift, the identity formulas and the emission law.

From them, three ids, and no two of them are ever equal:

  • Profile::chunking_id, over the chunking preimage, is the third field of every node identity. A node is addressed by the law that cut it, so declaring a canon base or renaming a predicate leaves every unit and section where it was and the two graphs merge.
  • Profile::contract_id is the law id, over the emission preimage, and it is what the graph states as sliceProfile. Two runs that agree on it agree on every byte they emit — which is why the canon base is inside it: a declared base turns every citation object from a typed literal into a minted IRI.
  • Profile::purremb_chunking_stage carries the chunking preimage as a PURREMB stage, whose canonical form is TLV-framed and which an embedding family therefore can reproduce — the id derived from it is the one a chunk target carries. Neither of the other two is a chunking-stage id and no family can derive them; a consumer that wires one into a family has named an id nothing on the other side can check.

The split is what a consumer feels. Embeddings are addressed by the chunking id, so they survive a renamed term and a newly declared canon base and are invalidated exactly when a unit’s bytes move.

Unit::text_chunk_target hands a unit over as the kernel’s own chunk subject, and verify_unit proves a unit against bytes by delegating to the kernel’s reference verification law, so this crate and a .purremb consumer refuse the same bytes for the same reason. Document::byte_to_scalar and Document::scalar_to_byte convert between the bytes the identities are in and the scalars a host indexes text by.

§It mints no vocabulary

PurRDF is not an ontology. Every class, predicate, and datatype IRI this crate emits, and the base under which it mints node IRIs, is caller-supplied configuration in a Vocabulary; there is no default namespace and no fabricated fallback. Vocabulary::under derives a whole vocabulary from one base IRI with the crate’s local names, and a caller who wants other IRIs sets the fields directly. Vocabulary::standard derives it from the one namespace the specification designates for documents meant to be exchanged — an opinion about interchange, asked for by name, never reached for on a caller’s behalf. The vocabulary is part of the profile’s identity: the same law under two vocabularies is two profiles, and the local names are inside it beside the base, so a renamed term is a law that says it changed.

No derived IRI wears two roles. The classes, the predicates and the datatypes are three disjoint sets — a heading’s text is carried by heading and typed with headingText — because a namespace offered for interchange has to be describable, and no OWL or SHACL document can declare one IRI both an owl:DatatypeProperty and an rdfs:Datatype. A caller setting the fields itself is sovereign over its own terms and may spend fewer of them.

Caller-supplied is not unchecked. Every IRI a caller states — each field of the vocabulary, the source id, the canon base, and every IRI a concordance anchor mints under that base — must be an absolute IRI under the workspace IRI law, which this crate reaches through purrdf-core so that it is asking the same question the kernel will ask when it interns the result. A relative IRI would otherwise slice whole and be refused a stage later, far from the field that wrote it.

That question has two halves and they are refused apart. A string the law reads whole and finds scheme-less is relative, and the remedy is to write it absolute. A string the law cannot read at all is malformed, and the refusal carries the law’s own finding — which byte, and what about it — because this crate does not own that law and has no business rewording it. https://example.org/% is a truncated percent-encoding, not an id “with no scheme”.

Re-exports§

pub use syntax::AuthoringDocument;
pub use syntax::CodeSpanLimits;
pub use syntax::SyntaxDocument;
pub use syntax::SyntaxError;
pub use syntax::SyntaxLimits;
pub use syntax::authoring;
pub use syntax::replace_code_spans;
pub use syntax::syntax;

Modules§

syntax
Bounded source syntax shared by authoring consumers and the RDF slicer.

Structs§

Citation
One concordance row, unflattened: the verse range it covers, its canon sources, its anchors, and what it lifted in this document.
Claim
One node’s triples, rendered as sorted N-Triples lines (valid Turtle; declare text/turtle to a consumer).
ContentAnchor
A unit’s text with bounded context on each side: the anchor a consumer re-finds the unit by after the document has been edited.
Document
A document as a typed layer of annotations over untouched source bytes: the stand-off model analyze returns.
MalformedRow
A row of a concordance table that could not be read as a citation at all, and why.
Profile
The slicing profile: its name and version, its vocabulary, the byte bound and overlap that act only on an oversize unit, and the optional canon IRI base for concordance anchors.
Section
A section of the document: a heading, or a movement marker.
SourceDocument
One source document: its IRI and its exact bytes.
Span
A byte range of the source: start inclusive, end exclusive.
Unit
A unit: a numbered verse, a paragraph, or one piece of either after the oversize split.
VerbatimSpan
One text-carrying span of a cover: the bytes at byte_start..byte_end, quoted verbatim.
Vocabulary
Every IRI the slicer emits, supplied by the caller. The fields are public so a caller can set any of them; Vocabulary::under fills all of them from one base, Vocabulary::standard fills them from the namespace the specification designates, and Vocabulary::validate is applied before any claim is rendered.

Enums§

ClaimKind
Which node a claim describes.
DecodeError
Why a graph could not be read back into a cover: the refusals of the graph half of the decode law. The span half’s refusals arrive wrapped, in DecodeError::Reconstruct.
MarkdownError
Why a document could not be sliced.
ReconstructError
Why a cover could not be decoded back into a document.
RowDefect
What a concordance row failed at.
SpanRelation
The thirteen ways two byte spans can lie against each other: Allen’s interval relations, with the six inverses named as their own variants.

Constants§

CONTEXT_BYTES
How many bytes of context a unit’s content anchor reaches on each side, snapped to a scalar boundary so the context is always text.
DEFAULT_MAX_BYTES
The default byte bound: a unit over this many bytes splits.
DEFAULT_OVERLAP
The default overlap: the continuation of a split reaches back this many bytes, snapped backward to a newline.
DIGEST_ALGORITHM
The digest algorithm tag carried inside every unit and section identity, in both the preimage and the IRI.
MIN_MAX_BYTES
The least byte bound a profile may declare: the widest UTF-8 scalar is four bytes, and no piece is ever over the bound, so a smaller bound could not be honoured without cutting inside a scalar.
PURREMB_PARAMETER_ENCODING
The parameter encoding Profile::purremb_chunking_stage declares: the stage’s parameters are exactly the bytes of Profile::chunking_bytes, which are UTF-8 plain text, one fact of the law per line.
RDF_REIFIES
rdf:reifies, the RDF 1.2 predicate binding a citation node to the triple term it reifies. Like RDF_TYPE it is the standard’s IRI and never a vocabulary field: reification is a shape of the data model, not a term a deployment gets to rename. rdf:reifies — the RDF 1.2 predicate linking a reifier to the triple term it reifies.
RDF_TYPE
rdf:type, one of the four IRIs the slicer emits that are the standard’s, not the caller’s. rdf:type — the instance-of predicate.
STANDARD_NAMESPACE
The namespace the specification designates for this crate’s terms: the one namespace Vocabulary::standard derives everything under.
XSD_HEX_BINARY
xsd:hexBinary, the datatype of a unit’s content digest stated as data. The lexical form is the same lowercase hex the unit’s IRI carries, so a consumer compares the two without re-encoding either. xsd:hexBinary — a sequence of hex-encoded bytes.
XSD_INTEGER
xsd:integer, the datatype of every ordinal, level, byte offset, and scalar offset. xsd:integer — arbitrary-magnitude (this crate: i128-bounded) signed integer.

Functions§

analyze
Reads a Markdown document into the typed stand-off model under a profile: the seam every refusal is stated at.
analyze_syntax
Project an already admitted syntax model without parsing Markdown again. Source bytes come from that model, so ranges cannot be paired with another input.
analyze_with_limits
Analyze through the shared native syntax scanner with explicit caller limits. The source-syntax model can also be retained and projected with analyze_syntax.
citation_iri
The identity of one citation edge: what a concordance row lifted onto one unit.
decode_document
The document back from its graph: the specification’s two extraction rules over a parsed dataset, then the kernel’s reconstruct over what they extracted.
reconstruct
The document back from its cover: writes each span’s bytes at its offset, requires every byte covered, admits an overlap only where a continues edge declares it and its bytes agree, and proves the result against the stated source digest.
render
Projects an analyzed document into claims under the law it was admitted under: Document::profile, and no other.
section_iri
The identity of a section, over its heading line’s bytes, under the same chunking id unit_iri takes.
slice_markdown
Slices a Markdown document into claims under a profile: analyze then render.
span_relation
The Allen interval relation of a to b, over half-open byte spans.
structure_iri
The identity of a structure node, over its own bytes, under the same chunking id and the same seven-field preimage unit_iri takes — only the kind differs.
unit_iri
The identity of a unit, with the digest algorithm inside both the preimage and the IRI. A consumer re-derives it from the bytes at a span to prove a hit is bound to its source.
verify_unit
Proves a unit against the bytes it claims to annotate: the span is inside them, it falls on scalar boundaries at both ends, and the bytes at it still digest to the digest the unit carries.