Expand description
purrdf-lex — the lexical foundations shared by every grammar in the
PurRDF workspace.
A pure-Rust, wasm-clean crate whose one runtime dependency is the
zero-dependency purrdf_hash root (its hex-digit reader). Every reader and
writer above it — the IRI parser, the Turtle/TriG/N-Triples/N-Quads codecs,
the SPARQL lexer, the SPARQL results writers, JSON-LD, JSON Schema and the
GTS container — decides token boundaries and escapes with the code here, so
each lexical law is written down once and every grammar agrees with every
other.
§Scope
This crate is the home of the workspace’s lexical layer: byte-class scanning, grammar terminals, term syntax, literal and IRI escaping, percent encoding, JSON strings, JSON pointers, a JSON reader and writer, a YAML reader and writer, a CBOR codec, an XML reader and Unicode normalisation, and the stack-free walks and ordered pair lists those readers and writers are built on. Each is a law a grammar states; none is a vocabulary, and nothing here mints an IRI.
- Grammar terminals —
terminals, the exact Turtle/SPARQL character classes (WS,PN_CHARS_BASE,PN_CHARS_U,PN_CHARS,VARNAME), the XML 1.0Char,NameStartCharandNameCharclasses, the UnicodeWhite_Spaceproperty, and the ECMA-262LineTerminatorand\sclasses, each a range table proved sorted and disjoint at compile time and answered below U+0100 by one class-table load. A scanner’s character class decides token BOUNDARIES under maximal munch, so an approximation misparses documents rather than merely widening the accepted language; one transcription is the only way to keep the scanners agreeing with each other. - Byte-class scanning —
scan, the chunked scanners built from those tables (terminals::find_first_trivia,terminals::find_first_iri_body_special,terminals::find_first_json_string_special,terminals::find_first_xml_special) andterminals::ByteClass, the same kernel over a caller’s own class. Each finds the first byte of a class sixteen bytes at a time, as run compares with no data-dependent branch: the shape LLVM lowers to packed byte compares and a mask extraction on x86_64, aarch64 and wasmsimd128, and to straight-line scalar code where the target has no vector unit.scan::find_byteandscan::find_byte2are the same kernel for a needle known only at run time. - Escape decoding —
terminals::decode_uchar,terminals::echar_valueandterminals::decode_char_ref: theUCHAR,ECHARand XMLCharRefdecoders every RDF, SPARQL, ShEx and XML reader shares, with every digit read bypurrdf_hash::hex::nibble, so a sign is never a digit. - Grammar whitespace —
terminals::skip_wsandterminals::trim_ws: theWS/ XMLS/ JSONwsskip and trim, four scalars and never FORM FEED. - JSON strings —
json_escape, the one RFC 8259 §7 string body escaper every PurRDF JSON writer shares, over the JSON string-body scanner (json_escape::JsonEscapesnames the spellings those writers pin), and the one decoder every reader shares (json_escape::unescape), which refuses an unpaired surrogate. - Term syntax and escaping —
term_syntax, the RDF 1.2 canonical spelling of an IRI, a literal, a blank node and the triple-term delimiters;iri_escape, which scalars anIRIREFwriter escapes and the emission; andliteral_escape, the one literal-body escaper, over the carriers a literal travels in (literal_escape::Carrier). Each writes into atext_out::TextOut, the append target every serializer shares. - XML —
xml, the one XML 1.0 + Namespaces reader: a pull reader and a document tree, with DOCTYPE and external entities refused, an explicit depth cap, no machine-stack recursion, and byte offsets in every error. - Constructors —
constructors!, the one spelling of an inherent constructor whose whole body is a conversion: a caller’s text taken asimpl Into<String>into a variant or a newtype field (constructors!), anewthat is the type’sDefault, and aFromimpl that wraps a lower layer’s value into a variant (variant_from!). - Crockford Base32 —
crockford, the text form of a 128-bit ULID. - JSON documents —
json, the one RFC 8259 reader, value and writer: a pulljson::Readerwith byte offsets, streaming skips and an occurrence table, ajson::Valuewhose numbers keep their lexemes and whose objects keep every member in order, and a deterministic compact and pretty writer, none of them recursing on the machine stack. - YAML documents —
yaml, the one YAML 1.2 reader and deterministic block emitter, over the JSON data model. - CBOR —
cbor, the one RFC 8949 codec: the shortest-form head, as-is and core deterministic encoding, and well-formed and deterministic decoding. - JSON Pointer —
json_pointer, RFC 6901 reference tokens. - Percent-encoding —
percent, the RFC 3986 encoder over the sets the specifications define, the strict and form-urlencoded decoders, and RFC 3986 §6.2.2 normalization. - Unicode normalization —
unicode, the workspace’s one normalization pipeline: NFC, NFD, NFKC and NFKD (UAX 15), the NFC verdict, the canonical combining class, and the streaming stages a caller composes with its own (the full-text analyzer’s case fold), over tables generated from the vendored Unicode Character Database atunicode::UNICODE_VERSION, the one version every Unicode table in the workspace is generated from. - Deep trees —
walk, what every recursive type of the workspace walks itself with instead of the machine stack: the inline-first work list (walk::WorkList), the iteratively dropped box (walk::Nested) and the writer that prints such a type exactly as#[derive(Debug)]would (walk::write_debug). - Ordered pairs —
assoc, the first-match lookup and the replace-in-place-or-append insert of a[(K, V)]list whose order is part of its value (a JSON object’s members, a GTS graph’s metadata). - Message errors —
message_error!, the one declaration of an error type whose whole content is a human-readable message;error_source!declares the borrowed typed causes of an enum’s wrapper variants. - Diagnostic presentations —
diagnostic, a diagnostic’s stable message identity and exact typed arguments (diagnostic::DiagnosticPresentation), checked against its English template and rendered once, with a JSON record whose integers survive binary64 hosts.
§Examples
Find where a run of Turtle/SPARQL whitespace ends, test a terminal, and escape a JSON string body:
use purrdf_lex::json_escape::{JsonEscapes, push_string};
use purrdf_lex::terminals::{find_first_trivia, is_pn_chars_base};
assert_eq!(find_first_trivia(b" \t\r\n?s"), Some(4));
assert!(is_pn_chars_base('é'));
assert!(!is_pn_chars_base('_'));
let mut out = String::new();
push_string(&mut out, "say \"hi\"\n", JsonEscapes::Minimal);
assert_eq!(out, r#""say \"hi\"\n""#);Modules§
- allocation
- Fallible physical buffer growth for native lexical parsers and compilers.
- assoc
- Ordered association lists: a
[(K, V)]whose order is part of the value, read by FIRST match and written by replacing the first match in place or appending. - cbor
- The workspace’s one CBOR codec (RFC 8949).
- constructors
- The inherent constructors whose whole body is one conversion.
- control
- Preserve native refusals and caller progress failures without type erasure.
- crockford
- Crockford Base32 for 128-bit values: the text form of a ULID.
- diagnostic
- Typed diagnostic presentation, exact JSON arguments and validated rendering.
- error
- Borrowed typed error inspection shared by all native producers. Borrowed error inspection, independent of human presentation.
- gb18030
- Explicit, strict GB 18030 editions with incremental source contributions.
- html
- HTML character references, independently implemented from the WHATWG reference submachine, borrowed tag/attribute tokenization and opaque HTML forms. None normalizes line endings or recursively decodes its output.
- integer_
text - Checked unsigned digit conversion and borrowed integer-word operations.
- iri_
escape - The ONE transcription of which scalars an
IRIREFwriter must escape — the egress mirror ofcrate::terminals::is_iriref_forbidden. - json
- The workspace’s one JSON reader, value and writer (RFC 8259).
- json_
escape - The workspace’s one JSON string-body escape law (RFC 8259 §7).
- json_
pointer - JSON Pointer (RFC 6901): the reference-token escape, pointer parsing and the array-index token.
- literal_
escape - The one escaper for the body of an RDF string literal — the text between the quotes of an N-Triples, N-Quads, Turtle, TriG or SPARQL literal.
- percent
- Percent-encoding (RFC 3986 §2.1): the encoder over the sets the
specifications define, the strict decoder, the
application/x-www-form-urlencodeddecoder, and RFC 3986 §6.2.2 normalization. - scan
- Chunked byte-class scanners over the
terminalstables. - term_
syntax - The RDF 1.2 term syntax every text writer spells a term with: an IRI as an
escaped
IRIREF, a literal in its canonical N-Triples form, a blank node as aBLANK_NODE_LABEL, and the delimiters of a triple term. - terminals
- Exact character classes for the Turtle/SPARQL/XML terminal grammars — the ONE transcription every scanner in the workspace scans with.
- text_
out TextOut, the infallible append target every text writer in the workspace emits into.- unicode
- Unicode normalization: the four forms of
UAX #15, the canonical combining class, and the streaming stages they are built from, over tables generated from the vendored Unicode Character Database. - visual_
fingerprint - Admitted visual-hashing 1.0.0 presentations of authoritative content identities.
- walk
- Whole-tree walks that spend heap, never machine stack: the work list every
such walk keeps (
WorkList), and theDebugwriter (write_debug) that prints a recursive type exactly as#[derive(Debug)]would, one token at a time. - xml
- The workspace’s one XML reader: XML 1.0 (Fifth Edition) with Namespaces in
XML 1.0, as a pull
Readerand a read-onlyDocumenttree built from it. - yaml
- The workspace’s one YAML 1.2 grammar, lexical syntax and JSON projection.
Macros§
- constructors
- Inherent constructors whose whole body is one conversion.
- error_
source - Implement an enum error’s borrowed cause through its named wrapper variants.
- json_
record - Write a record type’s
ToJsonandFromJsonfrom one member list. - json_
string_ enum - Spell each variant of a closed, field-less enum as one JSON string.
- message_
error - Declare an error type whose whole content is one human-readable message.
- resident_
clone - Implement resident
Clonethrough a type’s original admitted clone body. Native callers useclone_with_memorydirectly and retain its admission. - variant_
from Fromimpls that wrap a source value into one variant of the target.