Skip to main content

Crate purrdf_text

Crate purrdf_text 

Source
Expand description

purrdf-text — deterministic full-text search over RDF 1.2 literals.

This crate builds an in-memory inverted index over the literals of a frozen purrdf-core dataset and answers ranked retrieval queries against it. Text is put through one explicit resolved analyzer: compatibility caseless Unicode normalization, script-scoped accents, costed dictionary segmentation and optional English stemming. Atomic emoji, pre-stem surface words and punctuation-bearing substring spans retain original-source evidence. The standard law requires five caller-supplied baseline lexicon artifacts; an empty lexicon is an explicit profile. See Analyzer and AnalyzerProfile.

The index side and the query side run that same pipeline. They have to: a needle is matched against the dictionary by equality, so two pipelines that merely resemble each other produce a search that returns nothing and reports nothing.

§Explicit graph-free corpora

GraphLanguage remains the default: documents are keyed by graph, subject and language, and partitions by graph and language. GraphFree instead stores subject/language documents and language-only partitions. The graph coordinate occupies no storage and contributes no field to its separately domain-separated index and source identities. Both profiles use the same analyzer, scorer, token dictionary and ranked/occurrence relation code. Selecting a graph-free profile is explicit; an all-default quad dataset is not automatically reinterpreted as one. Open a TripleRead from its immutable triple carrier:

use purrdf_core::{RdfLiteral, TermValue, TripleDatasetBuilder};
use purrdf_text::{Analyzer, GraphFree, TextIndex, TextIndexConfig};
let mut builder = TripleDatasetBuilder::default();
let subject = builder.intern_iri("urn:document")?;
let predicate = builder.intern_iri("urn:text")?;
let text = builder.intern_literal(RdfLiteral::simple("river bank"))?;
builder.push_triple(subject, predicate, text)?;
let dataset = builder.freeze()?;
let read = dataset.read()?;
let config = TextIndexConfig::<GraphFree>::for_triples(
    vec![TermValue::iri("urn:text")], Analyzer::empty_lexicon(),
)?;
let index = TextIndex::from_dataset(&read, &config)?;
assert_eq!(index.document_count(), 1);

Graph-free construction still reads the original RDF 1.2 annotation layer, scoped identities and language/direction law. Named graph requests through a triple read refuse before missing-name shortcuts and retain the original operation failure; a fresh read of the same frozen carrier remains healthy.

§It mints no vocabulary

Ranked retrieval reaches SPARQL through the evaluator’s property-function seam, and the predicate IRIs a query calls it by are caller-supplied configuration. PurRDF is not an ontology: there is no namespace of this project’s own, no default IRI, and no fabricated fallback for a caller who supplies none. A configuration with no IRI is a TextError::Config, not a guess — the same rule that governs every other configurable vocabulary in this workspace. Which predicates’ literals are indexed is likewise the caller’s decision.

§Every score is exact, and identical on every target

Ranking is done entirely in base-10 fixed-point integer arithmetic (Fixed, SCALE_DIGITS fractional digits). No floating-point value enters this crate; the crate root denies clippy::float_arithmetic, so none can.

That is a correctness requirement rather than a preference. BM25 needs a natural logarithm, and a libm ln may differ by a unit in the last place between implementations — which is enough to reverse the order of two near-tied documents. The same query over the same data would then return rows in one order from a native build and another from a wasm32-unknown-unknown build of the same engine: an answer divergence, and one nothing downstream could detect. Fixed::ln is instead a fixed-length integer series — a fixed iteration count, never a convergence test — so its result is a pure function of its input on every target.

Native Rust pins the ranking — row order and every score’s decimal lexical — against independent expectations. It also asserts byte identity of the serialized answer from two independently built indexes queried through the property-function seam. make wasm separately builds this release crate; WASM execution is reserved for actual target paths and host interfaces.

dictionary compiles complete authored phrase dictionaries with this same resolved analyzer, and PhraseOccurrenceRelation publishes their exact positional rows under an explicit caller IRI. The original token relation also advertises the dictionary physical capability to the one SPARQL planner. Dictionary caches retain their original compilation owners; matches bind to the current immutable index rather than cached occurrence IDs.

Re-exports§

pub use character::HanCharacterIndex;
pub use character::HanMatchEvidence;

Modules§

character
Independent Han-character retrieval over the same source-aligned analysis.
dictionary
Analyzer-bound dictionary matching over the index’s original token streams. All matches preserve the ordinary positional join’s ordered bag. Greedy longest-first selection is an explicit logical consumer, not a shape flag.
phonetic
Latin phonetic candidates and exact bounded scalar Levenshtein distance.
segment
Immutable dictionaries and integer weighted lattice segmentation.
stem
English Snowball 3.1.0 over scalar positions and the pinned Latin script.
unicode
The analyzer’s Unicode layer: full case folding, the four normalization forms, and word and sentence boundaries, over tables generated from the vendored Unicode Character Database.

Structs§

AccentScripts
Any subset of the five supported accent-fold scripts.
AlignedText
Normalized bytes and their actual original UTF-8 contributors.
Analysis
All views produced atomically by one analysis pass.
Analyzer
Immutable resolved analysis law. No filesystem, network or hidden data access.
AnalyzerProfile
A validated analysis law, independent of artifact storage representation.
AnalyzerScratch
Reusable lexical normalization, segmentation and word storage.
BoundedScore
A pure scorer’s value and its original query/corpus certificate.
Document
One indexed document: the text of one (graph, subject, language) triple key.
FieldEnvelope
Raw extrema over actual posting records in one field. Providers retain these facts with their posting blocks; they contain no cached corpus normalization.
FieldInput
Exact document facts for one field. These are validated even when the frequency or weight is zero.
Fixed
A base-10 fixed-point number: the value is the raw integer divided by ten raised to SCALE_DIGITS.
GraphFree
Explicit subject/language documents and language-only partitions. Its graph coordinate and selector occupy zero bytes and contribute no digest field.
GraphLanguage
The existing graph/subject/language document and graph/language partition law.
IndexPositions
The resident index’s owned positional cursor state.
IndexRankedPostings
Owned native ranked cursor. Canonical ranks are resolved against the pinned snapshot; retained summaries themselves never persist shifting ranks.
PartitionFilter
Which partitions a query is restricted to, before any ranking happens.
PartitionKey
The (graph, language) pair whose corpus statistics are computed together.
PartitionStats
One partition’s corpus statistics.
PhoneticMatch
Shared pronunciation codes refined by exact spelling distance.
PhraseOccurrenceRelation
?doc caller:phrase(?phrase ?language ?start ?end), with half-open positions. The IRI is supplied by the caller. The existing single-token relation is unchanged.
PreparedCorpus
Validated corpus statistics and their ranking profile.
PreparedQuery
Prepared, corpus-bound term statistics. IDFs are computed once per query term and corpus, never accepted from a caller or recomputed per document.
Projection
One independently bounded projection with inspectable source evidence.
RankedWindow
One unread posting head and corpus-independent bounds for its physical block and remaining partition suffix. All vectors retain the supplied admission.
RankingField
One field’s named, immutable scoring parameters.
RankingProfile
A validated ranking law, separate from tokenization and stored postings.
ScopedTerm
A query-time membership restriction on one analyzed token.
ScoreBound
A certified inclusive raw-score bound for one prepared query and corpus.
Scored
One ranked document.
SearchObservations
What TextSearchRelation’s invocations actually did, counted at the two places the difference between a point lookup and a ranking is decided.
SourceCoverage
Which of a configuration’s predicates the dataset carried a statement for.
SourceMatchEvidence
One occurrence traced to its original literal.
SubstringLimits
Immutable logical work limits, included in the analyzer profile identity.
SubstringMatch
Exact overlapping substring occurrences and original-source evidence.
SubstringRefusal
A complete typed refusal; it never carries partial matches.
SubstringReport
Complete matches and the logical work used to establish them.
SubstringWork
Charged logical work; independent of CPU, search strategy and architecture. Charges precede work, so a refusal includes the first charge exceeding its budget. Exact byte verification never starts before selective admission.
SurfaceIndex
Persistent scalar-gram and pronunciation indexes over immutable sources.
SurfaceTerm
A borrowed normalized term and current sorted document membership.
TermContribution
One needle term’s share of one document’s score.
TermOccurrenceRelation
Positional matching over an immutable TextSource, as a SPARQL relation.
TextIndex
A built, immutable inverted index over one dataset’s literals.
TextIndexConfig
The caller’s complete, dataset-independent statement of what to index.
TextSearchRelation
Ranked retrieval over an immutable TextSource, as a SPARQL relation.
TextSourceBounds
Exact native relation bounds prepared once for one immutable source generation. Providers can retain this value with their persistent snapshot and return it from TextSource::relation_bounds without walking postings at registration.
Token
A lexical token and its consecutive ordinal.
TokenOccurrence
A retained token occurrence in original analyzed position coordinates. This is a derived index view; strings remain in the sole term dictionary.
UnicodeVersion
Unicode table version.
UnicodeVersions
Pinned versions of each Unicode stage.

Enums§

AccentFold
Script-scoped removal of nonspacing accent marks.
CapacityFailure
The workspace capability’s allocation-free stop classes, preserved by TEXT.
Constraint
How a query constrains one dimension of a PartitionKey.
GraphSelector
Which graph a dataset-independent configuration reads: the owned, value space selector an index configuration writes down once and applies to any dataset.
InputMode
Literal interpretation before Unicode analysis.
MatchProjection
The projection in which match offsets are expressed.
Segmentation
Dictionary selection; baseline artifacts are always caller-supplied bytes.
Stemming
Explicit stemming language.
SubstringRefusalReason
Why an indexed or explicitly exhaustive query did not produce an answer.
TextDeltaError
A refused index delta, retaining the original operands independently of its human-readable presentation.
TextError
A failure raised by the text index.

Constants§

ANALYZER_PROFILE_ID
Profiled analyzer identity. Its fingerprint binds every stage, bound, dictionary and Unicode version across lexical and surface retrieval.
B
Default field length normalization for the explicit single-field profile.
FINGERPRINT_BYTES
The width of a term fingerprint, in bytes — blake3’s default digest length.
INDEX_CORPUS_PROFILE_ID
Corpus construction used by the in-memory index: documents are (graph, subject, language), partitions are (graph, language), direction is merged, and zero-token documents are excluded. External stores provide their own already-partitioned counts to the pure prepared scorer.
K1
The fixed BM25F term-frequency saturation constant, k1 = 1.2.
MAX_TOKEN_SCALARS
Largest admitted nominal token bound.
RANKING_PROFILE_ID
The arithmetic and query aggregation contract, independent of field choices.
RANKING_PROFILE_VERSION
Revision of the complete ranking law, including intermediate rounding.
SCALE_DIGITS
How many base-10 fractional digits a Fixed carries.

Traits§

CorpusProfile
A closed compile-time corpus law. Both shapes use the original analyzer, partition construction, scorer and retrieval machinery.
TextSource
A pinned source snapshot. Document identifiers and partition ordinals use the native canonical order, independently of the provider’s physical addresses.

Functions§

explain
Every needle term’s share of one document’s score, in sorted term order.
fingerprint_terms
A FINGERPRINT_BYTES-wide digest of terms, in the order given.
rank_partition
Rank one partition’s documents against needle.
select
Rank every partition filter admits, and emit the rows in (partition key ASC, per-partition rank ASC) order.
select_scoped
Select ranked rows after applying every declared token-field membership.
snapshot_fingerprint
Compute the native generation from the immutable inputs that determine text answers. Persistent providers use this same preimage with their maintained source multiset identity. Derived document tables and postings add no independent identity.
source_digest
The source digest and the coverage dataset yields under config — the two values an index built from that pairing holds (TextIndex::source_fingerprint and TextIndex::source_coverage).
source_fingerprint
Apply the native source domain to maintained selected literal-row hashes. Insert or remove the matching profile’s hash from purrdf_core::source::LiteralSourceRow::hashes_with_memory for each selected statement. Counts and membership belong to the original source owner; this multiset identity is not an authentication proof.
unicode_versions
Every table is generated from the same versioned Unicode data.
verify_binding
Check that index was built from dataset under config.