Module ingest 

Ingestion: parse RDF text (N-Triples / N-Quads / Turtle) and assemble a complete .rete file image. Shared by the CLI’s build/validate commands and the wasm bindings (the playground’s in-browser builder).

The N-Triples/N-Quads reader is line-based and keeps terms as their exact canonical token strings (<iri>, _:bnode, "lit", "lit"^^<dt>, "lit"@lang) so they double as dictionary keys and so a query can match by the same string. This is not a full RDF 1.1 parser — it covers the canonical N-Triples surface (and is deliberately tolerant of IRIs a strict parser would reject), enough for v0 ingestion. Turtle goes through oxttl.

That tolerance is measured, not silent: pass an IriAudit to any of the *_audited entry points and the parse counts every IRI outside the N-Triples IRIREF grammar / RFC 3987 (see crate::iri) — or, with strict, refuses the input on the first one. rete build always audits.

Assembly degrades with the build features: without the compression feature (e.g. on wasm) sections are written with the NONE codec — larger files, byte-compatible readers.

Structs

BuildStats

Counts describing an assembled file, for status lines and UIs.

DeferredMetadata

A metadata payload whose final form needs the written (deduplicated) counts. See IntoMetadata.

FinalCounts

The deduplicated counts of the file being written — what the header records, and therefore what any embedded metadata has to report.

IriAudit

Policy and tally for the invalid-IRI check that runs during ingest.

Enums

IngestError

Why an ingest failed. Parser errors from oxttl/oxrdfxml and I/O errors are flattened to their Display string rather than wrapped: the variants then carry no foreign types, which keeps the error usable from the wasm bindings and stable across upstream parser upgrades.

QuotedTripleSurface

Which surface a << … >> in Turtle/TriG input is read as — the input half of rete export --quoted-triple-syntax, same vocabulary, same two values.

Traits

IntoMetadata

What a build’s metadata callback may return.

Functions

assemble_dataset

Assemble a complete .rete file image from parsed quads: one shared dictionary, the default-graph index, one index per named graph, and the community pyramid. metadata is the opaque metadata-section payload (the CLI puts a JSON Dataset Card there); pass &[] for none — that is byte-identical to a metadata-free build.

assemble_dataset_streaming

Two-pass, low-RAM assembly: build a .rete by streaming the input(s) twice instead of holding every parsed quad in memory. stream is invoked twice and MUST replay the exact same quads in the same order each time — pass 1 observes every term into the dictionary; pass 2 encodes them to id-triples. The raw string quads (every term an owned String, heavily duplicated — by far the largest working set on a big graph) are never collected, so peak RAM is bounded by the dictionary + id-triples + index rather than the string-quad multiset. Output is byte-identical to assemble_dataset_with_opts on the same quads (same dictionary, same id-triples in file order, same downstream pipeline).

assemble_dataset_streaming_algo

Like assemble_dataset_streaming, but selects the community PyramidAlgo (the streaming, low-RAM build path for rete build --pyramid-algo …).

assemble_dataset_streaming_with_perms

Like assemble_dataset_streaming_algo, but writes only the permutations in perms — the two-pass low-RAM twin of assemble_dataset_with_perms.

assemble_dataset_with

Like assemble_dataset, but the metadata payload is derived from the BuildStats and the source quads while they are still resident — for metadata that describes the graph (the Dataset Card). Returning an empty Vec is byte-identical to a metadata-free build; return a DeferredMetadata for a payload that must also carry the file’s deduplicated counts, which exist only once the indexes are built.

assemble_dataset_with_opts

Like assemble_dataset_with, but with_pyramid = false skips the Louvain community pyramid entirely — no pyramid section is written (header length 0). SPARQL / SHACL / triple / reachability queries don’t use the pyramid, so a pyramid-less file is fully queryable and markedly smaller (the pyramid is the largest section on highly-connected graphs). Only the community / summary / progressive paths need it.

assemble_dataset_with_opts_algo

Like assemble_dataset_with_opts, but selects the community PyramidAlgo (the in-memory build path for rete build --pyramid-algo …).

assemble_dataset_with_perms

Like assemble_dataset_with_opts_algo, but writes only the permutations in perms (rete build --permutations 3). Every other byte of the file is unchanged; crate::index::PermSet::ALL reproduces the default build exactly, down to the header byte.

parse

Parse N-Triples text into raw term-token triples, skipping blank/comment lines.

parse_audited

parse with an optional IriAudit.

parse_quads

Parse N-Quads text: subject predicate object [graph] . per line.

parse_quads_audited

parse_quads with an optional IriAudit.

parse_rdfxml

Parse RDF/XML into canonical N-Triples-token triples via oxrdfxml. This is how most OWL ontologies ship (.rdf/.owl/.xml with an rdf:RDF root) — so rete ingests them directly, no external conversion. (OWL/XML — the non-RDF functional XML serialization — is a different language; convert it with owlready2 first.)

parse_reader

Stream-parse N-Triples ("nt") or N-Quads ("nq") from a reader, one line at a time, so the whole input text is never resident — the big-build memory win over reading the file into a String first (each line String is transient, freed every iteration). cap pre-sizes the output Vec (e.g. file_len / 64) to avoid reallocation doublings of the large spine. Turtle is not streamable here (oxttl needs the whole input); callers use the text path for "ttl".

parse_reader_audited

parse_reader with an optional IriAudit.

parse_reader_audited_surface

parse_reader_audited reading << … >> in the given QuotedTripleSurface (Turtle/TriG only — see the enum).

parse_statements

Parse one text input by format name ("nt", "nq", "ttl", "trig", or "rdfxml") into quads (triples land in the default graph).

parse_statements_audited

parse_statements with an optional IriAudit. The line-based syntaxes audit as they parse (so strict fails on the offending line); the others are audited over the parsed quads, which is where oxttl has already had its say.

parse_statements_audited_surface

parse_statements_audited reading << … >> in the given QuotedTripleSurface. This is the entry point rete build and rete validate use for everything but a streamed line-based input, and the one that turns “RDF 1.2 file, RDF-star reader” into an error that names the flag instead of a bare syntax complaint.

parse_trig

Parse TriG into canonical quads via oxttl — Turtle plus named-graph blocks (<g> { … }), the shape most large RDF dumps ship in. Reads << s p o >> as an RDF-star quoted triple; parse_trig_surface chooses.

parse_trig_surface

parse_trig in the given QuotedTripleSurface.

parse_turtle

Parse Turtle into canonical N-Triples-token triples via oxttl, reading << s p o >> as an RDF-star quoted triple. parse_turtle_surface chooses.

parse_turtle_surface

parse_turtle in the given QuotedTripleSurface.

stream_reader

Stream N-Triples ("nt") / N-Quads ("nq") from a reader, invoking f once per parsed quad — like parse_reader but without collecting into a Vec. Each quad’s term Strings are owned by f (and dropped when it returns if it doesn’t retain them), so a caller that only needs to observe every term — e.g. the two-pass assemble_dataset_streaming building its dictionary — never materializes the whole quad multiset. The big-graph, low-RAM ingest primitive. Blank/comment lines are skipped; a parse error stops the stream and is returned.

stream_reader_audited

stream_reader with an optional IriAudit.

stream_reader_audited_surface

stream_reader_audited reading << … >> in the given QuotedTripleSurface.

Type Aliases

RawQuad

A parsed quad: the triple plus an optional graph term (None = default graph).

RawTriple

A parsed triple as three canonical term tokens.