Struct Corpus 

pub struct Corpus { /* private fields */ }

Implementations

impl Corpus

pub fn from_projector(p: Box<dyn Projector>) -> Result<Corpus>

Build the index + forward store by draining any projector’s Situation stream.

pub fn from_csv(path: &Path) -> Result<Corpus>

pub fn from_json(path: &Path) -> Result<Corpus>

pub fn from_jsonl(path: &Path, max_lines: Option<usize>) -> Result<Corpus>

pub fn from_folder(dir: &Path) -> Result<(Corpus, FolderReport)>

Ingest a whole folder into ONE corpus — the “ask over a folder” path. Each file is auto-routed to a projector by extension (csv/tsv, json/ndjson/jsonl, and — with the onnx feature + models present — txt/md). Every situation is tagged with a src/<file> token and its file is shown, so the agent can slice by source. Unsupported files are skipped and counted.

pub fn from_document(path: &Path) -> Result<Corpus>

Single document (pdf/docx/pptx/html/md/txt) → extract text → text-projected corpus. Uses the env/default text engine; errors if the models or extractable text are absent.

pub fn from_text( path: &Path, ml_bundle: &Path, splade_dir: Option<&Path>, ) -> Result<Corpus, Box<dyn Error + Send + Sync>>

Native text projection (SPO tagger + optional English SPLADE) → queryable corpus.

pub fn new_incremental( source: impl Into<String>, columns: Vec<String>, kind: CorpusKind, ) -> Corpus

An empty corpus to grow incrementally (realtime/hot-model ingest): add_situation appends as documents are projected, and the query/agent layer sees each addition immediately.

pub fn set_gazetteer_overlay(&mut self, path: impl Into<PathBuf>)

Point the query-side gazetteer at a growing-gazetteer overlay file (entities learned at ingest). Must be called before the first query so the lazily-loaded gazetteer picks it up.

pub fn add_situation( &mut self, tokens: Vec<String>, display: Vec<String>, ) -> u32

Append one projected situation. Invalidates the high-DF noise cache (the corpus size changed), so registration recomputes on the next analytics call. Returns the new sid.

pub fn add_situation_num( &mut self, tokens: Vec<String>, display: Vec<String>, numbers: Vec<(String, f64)>, ) -> u32

Append one situation with numeric fields (realtime ingest of CSV/text carrying quantities).

pub fn add_situation_polar( &mut self, tokens: Vec<String>, display: Vec<String>, numbers: Vec<(String, f64)>, beliefs: Vec<(String, f32)>, ) -> u32

Append a situation carrying infon polarity for some of its tokens (paper §1.2, §4): each (token, i) records that assertion’s belief level, so Dempster-Shafer Bel/Pl and signed mass are computable over the bitmap. Tokens absent from beliefs default to +1 (asserted).

pub fn index(&self) -> &InfonIndex<RoarPostings>

The underlying inverted index, for callers that need the lower-level set algebra or the topology programs directly.

pub fn query(&self, ikl: &str, limit: usize) -> QueryOut

pub fn facet_tokens(&self, facet: &str, limit: usize) -> Vec<(String, usize)>

Top tokens under a facet (for agent vocabulary discovery).

pub fn facet_names(&self) -> Vec<String>

Distinct facet names (token prefixes) present in the corpus — the real schema, for validating DSL/tool arguments against what was actually extracted.

pub fn has_token(&self, token: &str) -> bool

True if token (a facet/value) exists in the index.

pub fn belief_interval(&self, token: &str) -> (f64, f64)

Dempster-Shafer belief interval [Bel, Pl] for a token over the whole corpus (paper §4.1).

pub fn signed_mass(&self, token: &str) -> f64

Net signed infon mass for a token over the whole corpus.

pub fn linter(&self) -> Linter

A linter over this corpus’s vocabulary — validates IKL atoms, suggests corrections, repairs syntax (paper §2, the compile-time boundary).

pub fn top_token_leaves(&self, limit: usize) -> Vec<String>

Distinct leaf values of the most-supported tokens — candidate terms for ontology discovery over the corpus’s own vocabulary.

pub fn noise_tokens(&self) -> &HashSet<String>

REGISTRATION — non-discriminative tokens present in > ~12% of situations: the coarse fragments (developers, reference…) that co-occur with everything and drown the analytics. Concept facets only; contextual facets (time/geo/qty/dur) and the specific gazetteer/entity values survive. Ported from query.ts registration() (71e1714). Computed once, then cached.

pub fn search_ranked( &self, tokens: &[String], limit: usize, ) -> Vec<(u32, usize, Vec<String>)>

Retrieve situations matching ANY of tokens, ranked by coverage (how many of the query tokens each situation contains) then sid. This makes free-text search return the most-relevant passages first instead of document order. Returns (sid, coverage, cells).

pub fn entity_link(&self, question: &str) -> Vec<String>

Entity-linking: resolve a question’s mentions (acronyms, content words, prefixes, bigrams) plus any gazetteer surfaces to the store’s PRECISE existing tokens — the tokens docs are actually indexed under, which the coarse SPLADE projection of a short question misses. Pure set-algebra linking (no lexical text search). Noise tokens are excluded. Ported from query.ts (17d6490).

pub fn breakdown(&self, anchor: &str, facet: &str, k: usize) -> Value

pub fn crosstab( &self, anchor: &str, facet_a: &str, facet_b: &str, k: usize, ) -> Value

pub fn rank(&self, facet: &str, k: usize) -> Value

pub fn cooccurs(&self, token: &str, k: usize) -> Value

pub fn s_path(&self, a: &str, b: &str, s: usize) -> Value

pub fn s_clusters(&self, s: usize, k: usize) -> Value

pub fn narrow(&self, scope: &[String], filters: &[String]) -> Value

pub fn stats(&self) -> Stats