Struct Semlith 

pub struct Semlith {
    pub quiet: bool,
    pub boundary: Boundary,
    /* private fields */
}

Fields

quiet: bool

Print model-download progress to stderr. Off for the MCP server, where stdout/stderr are a protocol channel.

boundary: Boundary

What this caller may index. The default is the command line’s: the deny-list, and no confinement.

Implementations

impl Semlith

pub fn open(dir: impl AsRef<Path>, model: Option<Model>) -> Result<Self>

Open (or create) a store in dir.

model is only honoured when the store is new; an existing store keeps the model it was built with, since vectors from two models are not comparable.

pub fn open_existing(dir: impl AsRef<Path>) -> Result<Self>

Open a store that must already be one.

Semlith::open creates what it is given, which is what index wants and the opposite of what every read command wants: a mistyped store directory becomes an empty store that answers every question with nothing, and when several stores are searched at once the others hide it completely.

pub fn refresh(&mut self) -> Result<()>

Reload the vector index if another process has replaced it since this one read it — semlith watch re-embedding while an agent holds an MCP server open.

Costs one SQLite read when nothing has changed, which is the case almost every time it is called.

pub fn model(&self) -> &Model

pub fn dim(&self) -> usize

pub fn len(&self) -> usize

How many vectors the store holds.

Answered from SQLite unless the vectors happen to be resident already, because the caller asking this is usually stats or a fleet totalling its members, and neither is worth loading an index for. The two agree: a file’s hash is committed only once its vectors are durable, so the chunks of hashed files are exactly the vectors on disk. Mid-run, in the process doing the writing, the resident index is ahead of that and is the truth.

pub fn is_empty(&self) -> bool

pub fn db(&self) -> &Connection

pub fn dir(&self) -> &Path

The store directory, for a caller that needs to take the store lock itself rather than for the length of one call.

pub fn shards(&self) -> Option<(usize, usize)>

How many shards the store’s vectors are split across, and how many of them may be resident at once. None for a store written before 0.7.0, whose single index is all or nothing.

pub fn evictions(&self) -> u64

Shards this store has put down to stay inside its memory budget. Above zero means the corpus has outgrown the budget and queries are paying to read shards back.

pub fn tokenizer(&self) -> Option<&Tokenizer>

The tokenizer this store’s model embeds with, once it has been loaded.

None before the first embedding, which is why the ledger labels every row with what counted it: a graph-only session never loads a model, and a row counted at four characters per token must never be added to one counted properly.

pub fn warm(&mut self) -> Result<()>

Pay the model-load and index-warmup cost up front, so the first query is not slower than the rest.

pub fn warm_index(&mut self) -> Result<()>

Warm the vector index without loading a model, for a caller that shares one loaded model across several stores.

pub fn index_paths( &mut self, roots: &[PathBuf], on_file: impl FnMut(&Path, IndexProgress), ) -> Result<IndexReport>

Walk roots, embed anything new or changed, and drop anything that has disappeared from disk.

pub fn index_paths_within( &mut self, roots: &[PathBuf], budget: Duration, on_file: impl FnMut(&Path, IndexProgress), ) -> Result<IndexReport>

Semlith::index_paths that gives up its remaining work when budget runs out, reporting what it did not reach in IndexReport::remaining.

For a caller whose own deadline is shorter than a corpus — an MCP tool call, where a client that waited too long declares the server hung. One file is always indexed however small the budget, so calling repeatedly always finishes: indexing is keyed on content hashes, so the next call walks past what this one completed rather than redoing it.

pub fn forget(&mut self, path: &Path) -> Result<(usize, usize)>

Remove a single file from the store. Returns (chunks, images).

pub fn scan(&self) -> Result<Vec<Finding>>

Every file this store holds that today’s rules would refuse.

The path for a store indexed before a rule widened. Both halves of the decision run here — the deny-list against the name, and the content table against the text the store is actually holding — so a finding is exactly what an index run today would refuse. The CLI’s semlith scan and the portal’s Privacy page both call this one function; two implementations of “what should not be here” would eventually disagree, and the one that mattered would be whichever the user did not run.

pub fn forget_findings(&mut self, findings: &[Finding]) -> Result<usize>

Evict everything Semlith::scan found, and say how many files went.

pub fn search(&mut self, query: &str, k: usize) -> Result<Vec<Hit>>

Top-k chunks for query, best first, over the whole store.

pub fn matching_files(&self, filter: &Filter) -> Result<i64>

How many indexed files filter selects.

Zero is worth reporting on its own: a glob that matches nothing is a different problem from a corpus that does not discuss the query.

pub fn matching_paths(&self, filter: &Filter) -> Result<Vec<String>>

The indexed paths filter selects, in path order.

pub fn search_filtered( &mut self, query: &str, k: usize, filter: &Filter, ) -> Result<Vec<Hit>>

Top-k chunks for query within the part of the corpus filter selects, best first.

Both halves of the store are consulted: the vector index for meaning, FTS5 for the literal terms. Dense search alone reliably misses exact identifiers — an embedding of EMBED_BATCH is a point in the same neighbourhood as every other constant — which is precisely what someone grepping a codebase is looking for.

The filter is applied before each half picks its top-k, not after fusion. Post-filtering a global ranking returns almost nothing whenever the subset is a minority of the corpus, which is the case the filter exists for.

pub fn embed_query(&mut self, query: &str) -> Result<Vec<f32>>

The query, embedded with this store’s model: the vector Semlith::search_with_vector takes.

Public because a caller comparing two stores has to be able to hold the query vector still. Embedding the same text twice does not give the same floats — ONNX Runtime reduces across its threads in whatever order they finish — and under int8 quantisation that moves a ranking, which is noise in a measurement that is about something else.

pub fn search_with_vector( &mut self, query: &str, vector: &[f32], k: usize, filter: &Filter, ) -> Result<Vec<Hit>>

Semlith::search_filtered with the query already embedded.

This is what lets several stores share one loaded model: the caller embeds the query once per distinct model rather than once per store, so searching four stores built with the same model costs one embed and one resident copy of the weights.

vector must come from this store’s model. Vectors from two models are not comparable, and nothing downstream can tell.

pub fn search_ranked( &mut self, query: &str, vector: &[f32], k: usize, filter: &Filter, ) -> Result<Vec<(Hit, f32)>>

Semlith::search_with_vector, also returning each hit’s similarity to the query vector — 0.0 for a hit only the keyword half found.

The fused score is a sum of rank reciprocals, so two chunks that hold the same position in their own store’s ranking score identically. Within one store that is a rare tie between two chunks; across stores it is the normal case, because every store has a best hit whether or not it has an answer. The similarity is what tells those apart, so it leaves the store alongside the score rather than being thrown away here.

pub fn search_preferring( &mut self, query: &str, vector: &[f32], k: usize, filter: &Filter, prefer: Prefer, ) -> Result<Vec<(Hit, f32)>>

Semlith::search_ranked with the caller’s preference applied.

The query’s shape is read here rather than passed in, because it is a function of the query text and nothing else — every surface that wants to print it calls shape_of on the same string and gets the same answer, with no second source of truth to drift.

pub fn read(&self, target: &Target, filter: &Filter) -> Result<Option<Read>>

One span of one file, or the definitions to choose between.

Answers only from the store’s own chunks. Reading the file off disk would answer for content semlith never indexed and was never allowed to look at — the boundary that governs indexing has to govern reading, or an agent holding the agent key could read ~/.ssh/id_rsa by naming it as a span.

pub fn image_count(&self) -> Result<i64>

How many images this store holds.

pub fn stats(&self) -> Result<(i64, i64, i64)>

(files, chunks, indexed bytes)

pub fn save(&mut self) -> Result<()>

Write the index out durably, so an interrupted save cannot leave a half-written index behind.