Skip to main content

Crate purrdf_lex

Crate purrdf_lex 

Source
Expand description

purrdf-lex — the lexical foundations shared by every grammar in the PurRDF workspace.

A pure-Rust, wasm-clean crate whose one runtime dependency is the zero-dependency purrdf_hash root (its hex-digit reader). Every reader and writer above it — the IRI parser, the Turtle/TriG/N-Triples/N-Quads codecs, the SPARQL lexer, the SPARQL results writers, JSON-LD, JSON Schema and the GTS container — decides token boundaries and escapes with the code here, so each lexical law is written down once and every grammar agrees with every other.

§Scope

This crate is the home of the workspace’s lexical layer: byte-class scanning, grammar terminals, term syntax, literal and IRI escaping, percent encoding, JSON strings, JSON pointers, a JSON reader and writer, a YAML reader and writer, a CBOR codec, an XML reader and Unicode normalisation, and the stack-free walks and ordered pair lists those readers and writers are built on. Each is a law a grammar states; none is a vocabulary, and nothing here mints an IRI.

  • Grammar terminals — terminals, the exact Turtle/SPARQL character classes (WS, PN_CHARS_BASE, PN_CHARS_U, PN_CHARS, VARNAME), the XML 1.0 Char, NameStartChar and NameChar classes, the Unicode White_Space property, and the ECMA-262 LineTerminator and \s classes, each a range table proved sorted and disjoint at compile time and answered below U+0100 by one class-table load. A scanner’s character class decides token BOUNDARIES under maximal munch, so an approximation misparses documents rather than merely widening the accepted language; one transcription is the only way to keep the scanners agreeing with each other.
  • Byte-class scanning — scan, the chunked scanners built from those tables (terminals::find_first_trivia, terminals::find_first_iri_body_special, terminals::find_first_json_string_special, terminals::find_first_xml_special) and terminals::ByteClass, the same kernel over a caller’s own class. Each finds the first byte of a class sixteen bytes at a time, as run compares with no data-dependent branch: the shape LLVM lowers to packed byte compares and a mask extraction on x86_64, aarch64 and wasm simd128, and to straight-line scalar code where the target has no vector unit. scan::find_byte and scan::find_byte2 are the same kernel for a needle known only at run time.
  • Escape decoding — terminals::decode_uchar, terminals::echar_value and terminals::decode_char_ref: the UCHAR, ECHAR and XML CharRef decoders every RDF, SPARQL, ShEx and XML reader shares, with every digit read by purrdf_hash::hex::nibble, so a sign is never a digit.
  • Grammar whitespace — terminals::skip_ws and terminals::trim_ws: the WS / XML S / JSON ws skip and trim, four scalars and never FORM FEED.
  • JSON strings — json_escape, the one RFC 8259 §7 string body escaper every PurRDF JSON writer shares, over the JSON string-body scanner (json_escape::JsonEscapes names the spellings those writers pin), and the one decoder every reader shares (json_escape::unescape), which refuses an unpaired surrogate.
  • Term syntax and escaping — term_syntax, the RDF 1.2 canonical spelling of an IRI, a literal, a blank node and the triple-term delimiters; iri_escape, which scalars an IRIREF writer escapes and the emission; and literal_escape, the one literal-body escaper, over the carriers a literal travels in (literal_escape::Carrier). Each writes into a text_out::TextOut, the append target every serializer shares.
  • XML — xml, the one XML 1.0 + Namespaces reader: a pull reader and a document tree, with DOCTYPE and external entities refused, an explicit depth cap, no machine-stack recursion, and byte offsets in every error.
  • Constructors — constructors!, the one spelling of an inherent constructor whose whole body is a conversion: a caller’s text taken as impl Into<String> into a variant or a newtype field (constructors!), a new that is the type’s Default, and a From impl that wraps a lower layer’s value into a variant (variant_from!).
  • Crockford Base32 — crockford, the text form of a 128-bit ULID.
  • JSON documents — json, the one RFC 8259 reader, value and writer: a pull json::Reader with byte offsets, streaming skips and an occurrence table, a json::Value whose numbers keep their lexemes and whose objects keep every member in order, and a deterministic compact and pretty writer, none of them recursing on the machine stack.
  • YAML documents — yaml, the one YAML 1.2 reader and deterministic block emitter, over the JSON data model.
  • CBOR — cbor, the one RFC 8949 codec: the shortest-form head, as-is and core deterministic encoding, and well-formed and deterministic decoding.
  • JSON Pointer — json_pointer, RFC 6901 reference tokens.
  • Percent-encoding — percent, the RFC 3986 encoder over the sets the specifications define, the strict and form-urlencoded decoders, and RFC 3986 §6.2.2 normalization.
  • Unicode normalization — unicode, the workspace’s one normalization pipeline: NFC, NFD, NFKC and NFKD (UAX 15), the NFC verdict, the canonical combining class, and the streaming stages a caller composes with its own (the full-text analyzer’s case fold), over tables generated from the vendored Unicode Character Database at unicode::UNICODE_VERSION, the one version every Unicode table in the workspace is generated from.
  • Deep trees — walk, what every recursive type of the workspace walks itself with instead of the machine stack: the inline-first work list (walk::WorkList), the iteratively dropped box (walk::Nested) and the writer that prints such a type exactly as #[derive(Debug)] would (walk::write_debug).
  • Ordered pairs — assoc, the first-match lookup and the replace-in-place-or-append insert of a [(K, V)] list whose order is part of its value (a JSON object’s members, a GTS graph’s metadata).
  • Message errors — message_error!, the one declaration of an error type whose whole content is a human-readable message; error_source! declares the borrowed typed causes of an enum’s wrapper variants.
  • Diagnostic presentations — diagnostic, a diagnostic’s stable message identity and exact typed arguments (diagnostic::DiagnosticPresentation), checked against its English template and rendered once, with a JSON record whose integers survive binary64 hosts.

§Examples

Find where a run of Turtle/SPARQL whitespace ends, test a terminal, and escape a JSON string body:

use purrdf_lex::json_escape::{JsonEscapes, push_string};
use purrdf_lex::terminals::{find_first_trivia, is_pn_chars_base};

assert_eq!(find_first_trivia(b" \t\r\n?s"), Some(4));
assert!(is_pn_chars_base('é'));
assert!(!is_pn_chars_base('_'));

let mut out = String::new();
push_string(&mut out, "say \"hi\"\n", JsonEscapes::Minimal);
assert_eq!(out, r#""say \"hi\"\n""#);

Modules§

allocation
Fallible physical buffer growth for native lexical parsers and compilers.
assoc
Ordered association lists: a [(K, V)] whose order is part of the value, read by FIRST match and written by replacing the first match in place or appending.
cbor
The workspace’s one CBOR codec (RFC 8949).
constructors
The inherent constructors whose whole body is one conversion.
control
Preserve native refusals and caller progress failures without type erasure.
crockford
Crockford Base32 for 128-bit values: the text form of a ULID.
diagnostic
Typed diagnostic presentation, exact JSON arguments and validated rendering.
error
Borrowed typed error inspection shared by all native producers. Borrowed error inspection, independent of human presentation.
gb18030
Explicit, strict GB 18030 editions with incremental source contributions.
html
HTML character references, independently implemented from the WHATWG reference submachine, borrowed tag/attribute tokenization and opaque HTML forms. None normalizes line endings or recursively decodes its output.
integer_text
Checked unsigned digit conversion and borrowed integer-word operations.
iri_escape
The ONE transcription of which scalars an IRIREF writer must escape — the egress mirror of crate::terminals::is_iriref_forbidden.
json
The workspace’s one JSON reader, value and writer (RFC 8259).
json_escape
The workspace’s one JSON string-body escape law (RFC 8259 §7).
json_pointer
JSON Pointer (RFC 6901): the reference-token escape, pointer parsing and the array-index token.
literal_escape
The one escaper for the body of an RDF string literal — the text between the quotes of an N-Triples, N-Quads, Turtle, TriG or SPARQL literal.
percent
Percent-encoding (RFC 3986 §2.1): the encoder over the sets the specifications define, the strict decoder, the application/x-www-form-urlencoded decoder, and RFC 3986 §6.2.2 normalization.
scan
Chunked byte-class scanners over the terminals tables.
term_syntax
The RDF 1.2 term syntax every text writer spells a term with: an IRI as an escaped IRIREF, a literal in its canonical N-Triples form, a blank node as a BLANK_NODE_LABEL, and the delimiters of a triple term.
terminals
Exact character classes for the Turtle/SPARQL/XML terminal grammars — the ONE transcription every scanner in the workspace scans with.
text_out
TextOut, the infallible append target every text writer in the workspace emits into.
unicode
Unicode normalization: the four forms of UAX #15, the canonical combining class, and the streaming stages they are built from, over tables generated from the vendored Unicode Character Database.
visual_fingerprint
Admitted visual-hashing 1.0.0 presentations of authoritative content identities.
walk
Whole-tree walks that spend heap, never machine stack: the work list every such walk keeps (WorkList), and the Debug writer (write_debug) that prints a recursive type exactly as #[derive(Debug)] would, one token at a time.
xml
The workspace’s one XML reader: XML 1.0 (Fifth Edition) with Namespaces in XML 1.0, as a pull Reader and a read-only Document tree built from it.
yaml
The workspace’s one YAML 1.2 grammar, lexical syntax and JSON projection.

Macros§

constructors
Inherent constructors whose whole body is one conversion.
error_source
Implement an enum error’s borrowed cause through its named wrapper variants.
json_record
Write a record type’s ToJson and FromJson from one member list.
json_string_enum
Spell each variant of a closed, field-less enum as one JSON string.
message_error
Declare an error type whose whole content is one human-readable message.
resident_clone
Implement resident Clone through a type’s original admitted clone body. Native callers use clone_with_memory directly and retain its admission.
variant_from
From impls that wrap a source value into one variant of the target.