Module text 

Native text projection primitives (multilingual, CPU-only), ported from the TS ingest pipeline. • model2vec — the EN/JA/KO static span embedder (reuses the shipped potion.f32 artifact). • ot — Sinkhorn optimal transport + k-means codebook for auto ontology sensing. The SPLADE / SPO-tagger ONNX stages plug in next via the ort crate (ONNX Runtime), running the same exported .onnx graphs unchanged.

Re-exports

pub use gazetteer::GazHit;

pub use gazetteer::Gazetteer;

pub use model2vec::cosine;

pub use model2vec::Model2Vec;

pub use splade::FacetTerm;

pub use splade::SpladeProjector;

pub use tagger::Span;

pub use tagger::SpoTagger;

Modules

gazetteer

Gazetteer tier — deterministic HIGH-RESOLUTION token extraction. The SPLADE heads fire on single wordpieces, so multi-word entities (“Amazon EKS”, “Direct Connect Gateway”, “zero-ETL”) shatter into coarse fragments. This tier matches a closed vocabulary of surface→token entries (mined from the corpus + discovered facet examples) and emits the WHOLE-ENTITY facet/value token, so the bitmap carries discriminative tokens the analytics programs (rank/crosstab/structure) can reason over. Used symmetrically by ingest AND query so index and question share one high-res token space. Ported from src/ingest/gazetteer.ts (71e1714).

model2vec

model2vec static embedder — the MULTILINGUAL (EN/JA/KO) span embedder, ported from the TS Model2VecEmbedder. No forward pass: tokenize (no special tokens), look up each subword id in the packed potion.f32 matrix, mean-pool, L2-normalise. CPU-only, language-agnostic — the same artifact the ingest pipeline already ships.

ot

Entropy-regularised optimal transport (Sinkhorn–Knopp) + k-means codebook — the ontology-sensing numeric core, ported from src/ingest/spo/ot.ts (itself the port of python spo_sinkhorn.py). Vectors are assumed L2-normalised, so cosine == dot. This is what turns a cloud of span embeddings into a MECE facet codebook — no hand-authored schema.

splade

Multi-Facet Orthogonal SPLADE projector — runs the baked ONNX graph (encoder + K facet heads + SPLADE pool → vecs [1,K,V]) and reads the top active vocab terms per facet, emitting facet-prefixed infon tokens (actor/…, instrument/…, target/…, constraint/…). Ported from src/ingest/splade/index.ts.

tagger

SPO span tagger — runs the exported BERT token-classifier (spo.onnx) and BIO-decodes ENT / REL / QTY / GEO / TIME / IGNORE spans. Ported from src/ingest/spo/tagger.ts.