Expand description
purrdf-text — deterministic full-text search over RDF 1.2 literals.
This crate builds an in-memory inverted index over the literals of a frozen
purrdf-core dataset and answers ranked retrieval queries against it. Text
is put through one explicit resolved analyzer: compatibility caseless Unicode
normalization, script-scoped accents, costed dictionary segmentation and
optional English stemming. Atomic emoji, pre-stem surface words and
punctuation-bearing substring spans retain original-source evidence.
The standard law requires five caller-supplied baseline lexicon artifacts;
an empty lexicon is an explicit profile. See Analyzer and AnalyzerProfile.
The index side and the query side run that same pipeline. They have to: a needle is matched against the dictionary by equality, so two pipelines that merely resemble each other produce a search that returns nothing and reports nothing.
§Explicit graph-free corpora
GraphLanguage remains the default: documents are keyed by graph, subject
and language, and partitions by graph and language. GraphFree instead
stores subject/language documents and language-only partitions. The graph
coordinate occupies no storage and contributes no field to its separately
domain-separated index and source identities. Both profiles use the same
analyzer, scorer, token dictionary and ranked/occurrence relation code.
Selecting a graph-free profile is explicit; an all-default quad dataset is
not automatically reinterpreted as one. Open a
TripleRead from its immutable triple carrier:
use purrdf_core::{RdfLiteral, TermValue, TripleDatasetBuilder};
use purrdf_text::{Analyzer, GraphFree, TextIndex, TextIndexConfig};
let mut builder = TripleDatasetBuilder::default();
let subject = builder.intern_iri("urn:document")?;
let predicate = builder.intern_iri("urn:text")?;
let text = builder.intern_literal(RdfLiteral::simple("river bank"))?;
builder.push_triple(subject, predicate, text)?;
let dataset = builder.freeze()?;
let read = dataset.read()?;
let config = TextIndexConfig::<GraphFree>::for_triples(
vec![TermValue::iri("urn:text")], Analyzer::empty_lexicon(),
)?;
let index = TextIndex::from_dataset(&read, &config)?;
assert_eq!(index.document_count(), 1);Graph-free construction still reads the original RDF 1.2 annotation layer, scoped identities and language/direction law. Named graph requests through a triple read refuse before missing-name shortcuts and retain the original operation failure; a fresh read of the same frozen carrier remains healthy.
§It mints no vocabulary
Ranked retrieval reaches SPARQL through the evaluator’s property-function
seam, and the predicate IRIs a query calls it by are caller-supplied
configuration. PurRDF is not an ontology: there is no namespace of this
project’s own, no default IRI, and no fabricated fallback for a caller who
supplies none. A configuration with no IRI is a TextError::Config, not a
guess — the same rule that governs every other configurable vocabulary in
this workspace. Which predicates’ literals are indexed is likewise the
caller’s decision.
§Every score is exact, and identical on every target
Ranking is done entirely in base-10 fixed-point integer arithmetic
(Fixed, SCALE_DIGITS fractional digits). No floating-point value
enters this crate; the crate root denies clippy::float_arithmetic, so none
can.
That is a correctness requirement rather than a preference. BM25 needs a
natural logarithm, and a libm ln may differ by a unit in the last place
between implementations — which is enough to reverse the order of two
near-tied documents. The same query over the same data would then return rows
in one order from a native build and another from a
wasm32-unknown-unknown build of the same engine: an answer divergence, and
one nothing downstream could detect. Fixed::ln is instead a fixed-length
integer series — a fixed iteration count, never a convergence test — so its
result is a pure function of its input on every target.
Native Rust pins the ranking — row order and every score’s decimal lexical —
against independent expectations. It also asserts byte identity of the
serialized answer from two independently built indexes queried through the
property-function seam. make wasm separately builds this release crate;
WASM execution is reserved for actual target paths and host interfaces.
dictionary compiles complete authored phrase dictionaries with this same
resolved analyzer, and PhraseOccurrenceRelation publishes their exact
positional rows under an explicit caller IRI. The original token relation
also advertises the dictionary physical capability to the one SPARQL planner.
Dictionary caches retain their original compilation owners; matches bind to
the current immutable index rather than cached occurrence IDs.
Re-exports§
pub use character::HanCharacterIndex;pub use character::HanMatchEvidence;
Modules§
- character
- Independent Han-character retrieval over the same source-aligned analysis.
- dictionary
- Analyzer-bound dictionary matching over the index’s original token streams. All matches preserve the ordinary positional join’s ordered bag. Greedy longest-first selection is an explicit logical consumer, not a shape flag.
- phonetic
- Latin phonetic candidates and exact bounded scalar Levenshtein distance.
- segment
- Immutable dictionaries and integer weighted lattice segmentation.
- stem
- English Snowball 3.1.0 over scalar positions and the pinned Latin script.
- unicode
- The analyzer’s Unicode layer: full case folding, the four normalization forms, and word and sentence boundaries, over tables generated from the vendored Unicode Character Database.
Structs§
- Accent
Scripts - Any subset of the five supported accent-fold scripts.
- Aligned
Text - Normalized bytes and their actual original UTF-8 contributors.
- Analysis
- All views produced atomically by one analysis pass.
- Analyzer
- Immutable resolved analysis law. No filesystem, network or hidden data access.
- Analyzer
Profile - A validated analysis law, independent of artifact storage representation.
- Analyzer
Scratch - Reusable lexical normalization, segmentation and word storage.
- Bounded
Score - A pure scorer’s value and its original query/corpus certificate.
- Document
- One indexed document: the text of one
(graph, subject, language)triple key. - Field
Envelope - Raw extrema over actual posting records in one field. Providers retain these facts with their posting blocks; they contain no cached corpus normalization.
- Field
Input - Exact document facts for one field. These are validated even when the frequency or weight is zero.
- Fixed
- A base-10 fixed-point number: the value is the raw integer divided by ten
raised to
SCALE_DIGITS. - Graph
Free - Explicit subject/language documents and language-only partitions. Its graph coordinate and selector occupy zero bytes and contribute no digest field.
- Graph
Language - The existing graph/subject/language document and graph/language partition law.
- Index
Positions - The resident index’s owned positional cursor state.
- Index
Ranked Postings - Owned native ranked cursor. Canonical ranks are resolved against the pinned snapshot; retained summaries themselves never persist shifting ranks.
- Partition
Filter - Which partitions a query is restricted to, before any ranking happens.
- Partition
Key - The
(graph, language)pair whose corpus statistics are computed together. - Partition
Stats - One partition’s corpus statistics.
- Phonetic
Match - Shared pronunciation codes refined by exact spelling distance.
- Phrase
Occurrence Relation ?doc caller:phrase(?phrase ?language ?start ?end), with half-open positions. The IRI is supplied by the caller. The existing single-token relation is unchanged.- Prepared
Corpus - Validated corpus statistics and their ranking profile.
- Prepared
Query - Prepared, corpus-bound term statistics. IDFs are computed once per query term and corpus, never accepted from a caller or recomputed per document.
- Projection
- One independently bounded projection with inspectable source evidence.
- Ranked
Window - One unread posting head and corpus-independent bounds for its physical block and remaining partition suffix. All vectors retain the supplied admission.
- Ranking
Field - One field’s named, immutable scoring parameters.
- Ranking
Profile - A validated ranking law, separate from tokenization and stored postings.
- Scoped
Term - A query-time membership restriction on one analyzed token.
- Score
Bound - A certified inclusive raw-score bound for one prepared query and corpus.
- Scored
- One ranked document.
- Search
Observations - What
TextSearchRelation’s invocations actually did, counted at the two places the difference between a point lookup and a ranking is decided. - Source
Coverage - Which of a configuration’s predicates the dataset carried a statement for.
- Source
Match Evidence - One occurrence traced to its original literal.
- Substring
Limits - Immutable logical work limits, included in the analyzer profile identity.
- Substring
Match - Exact overlapping substring occurrences and original-source evidence.
- Substring
Refusal - A complete typed refusal; it never carries partial matches.
- Substring
Report - Complete matches and the logical work used to establish them.
- Substring
Work - Charged logical work; independent of CPU, search strategy and architecture. Charges precede work, so a refusal includes the first charge exceeding its budget. Exact byte verification never starts before selective admission.
- Surface
Index - Persistent scalar-gram and pronunciation indexes over immutable sources.
- Surface
Term - A borrowed normalized term and current sorted document membership.
- Term
Contribution - One needle term’s share of one document’s score.
- Term
Occurrence Relation - Positional matching over an immutable
TextSource, as a SPARQL relation. - Text
Index - A built, immutable inverted index over one dataset’s literals.
- Text
Index Config - The caller’s complete, dataset-independent statement of what to index.
- Text
Search Relation - Ranked retrieval over an immutable
TextSource, as a SPARQL relation. - Text
Source Bounds - Exact native relation bounds prepared once for one immutable source generation.
Providers can retain this value with their persistent snapshot and return it
from
TextSource::relation_boundswithout walking postings at registration. - Token
- A lexical token and its consecutive ordinal.
- Token
Occurrence - A retained token occurrence in original analyzed position coordinates. This is a derived index view; strings remain in the sole term dictionary.
- Unicode
Version - Unicode table version.
- Unicode
Versions - Pinned versions of each Unicode stage.
Enums§
- Accent
Fold - Script-scoped removal of nonspacing accent marks.
- Capacity
Failure - The workspace capability’s allocation-free stop classes, preserved by TEXT.
- Constraint
- How a query constrains one dimension of a
PartitionKey. - Graph
Selector - Which graph a dataset-independent configuration reads: the owned, value space selector an index configuration writes down once and applies to any dataset.
- Input
Mode - Literal interpretation before Unicode analysis.
- Match
Projection - The projection in which match offsets are expressed.
- Segmentation
- Dictionary selection; baseline artifacts are always caller-supplied bytes.
- Stemming
- Explicit stemming language.
- Substring
Refusal Reason - Why an indexed or explicitly exhaustive query did not produce an answer.
- Text
Delta Error - A refused index delta, retaining the original operands independently of its human-readable presentation.
- Text
Error - A failure raised by the text index.
Constants§
- ANALYZER_
PROFILE_ ID - Profiled analyzer identity. Its fingerprint binds every stage, bound, dictionary and Unicode version across lexical and surface retrieval.
- B
- Default field length normalization for the explicit single-field profile.
- FINGERPRINT_
BYTES - The width of a term fingerprint, in bytes —
blake3’s default digest length. - INDEX_
CORPUS_ PROFILE_ ID - Corpus construction used by the in-memory index: documents are
(graph, subject, language), partitions are(graph, language), direction is merged, and zero-token documents are excluded. External stores provide their own already-partitioned counts to the pure prepared scorer. - K1
- The fixed BM25F term-frequency saturation constant,
k1 = 1.2. - MAX_
TOKEN_ SCALARS - Largest admitted nominal token bound.
- RANKING_
PROFILE_ ID - The arithmetic and query aggregation contract, independent of field choices.
- RANKING_
PROFILE_ VERSION - Revision of the complete ranking law, including intermediate rounding.
- SCALE_
DIGITS - How many base-10 fractional digits a
Fixedcarries.
Traits§
- Corpus
Profile - A closed compile-time corpus law. Both shapes use the original analyzer, partition construction, scorer and retrieval machinery.
- Text
Source - A pinned source snapshot. Document identifiers and partition ordinals use the native canonical order, independently of the provider’s physical addresses.
Functions§
- explain
- Every needle term’s share of one document’s score, in sorted term order.
- fingerprint_
terms - A
FINGERPRINT_BYTES-wide digest ofterms, in the order given. - rank_
partition - Rank one partition’s documents against
needle. - select
- Rank every partition
filteradmits, and emit the rows in(partition key ASC, per-partition rank ASC)order. - select_
scoped - Select ranked rows after applying every declared token-field membership.
- snapshot_
fingerprint - Compute the native generation from the immutable inputs that determine text answers. Persistent providers use this same preimage with their maintained source multiset identity. Derived document tables and postings add no independent identity.
- source_
digest - The source digest and the coverage
datasetyields underconfig— the two values an index built from that pairing holds (TextIndex::source_fingerprintandTextIndex::source_coverage). - source_
fingerprint - Apply the native source domain to maintained selected literal-row hashes.
Insert or remove the matching profile’s hash from
purrdf_core::source::LiteralSourceRow::hashes_with_memoryfor each selected statement. Counts and membership belong to the original source owner; this multiset identity is not an authentication proof. - unicode_
versions - Every table is generated from the same versioned Unicode data.
- verify_
binding - Check that
indexwas built fromdatasetunderconfig.