Struct Store 

pub struct Store { /* private fields */ }

Implementations

impl Store

pub async fn open(location: &Url) -> Result<Self>

Open against a local filesystem URL or a remote one for which the caller has no extra options to pass (env vars suffice). CLI verbs that load [storage] from config should call Store::open_with_options instead so the same options flow into every dataset open and write.

pub fn lance_cache_bytes(&self) -> u64

Live byte size of the shared Lance session caches (index + metadata). Diagnostic only - walks the caches.

pub async fn open_with_options( location: &Url, storage_options: HashMap<String, String>, caps: RuntimeCaps, ) -> Result<Self>

Open with object-store options (S3 creds, region, endpoint, …) threaded through Lance verbatim. Keys are the standard object_store config names; pond does not parse them. Empty options + default caps is equivalent to Store::open. Cache caps come from the [runtime] config block via crate::substrate::RuntimeCaps.

pub async fn open_local(path: impl AsRef<Path>) -> Result<Self>

Convenience for tests and CLI verbs holding a &Path: wraps the path in a file://... URL via config::url_for_path before opening. Routes through Store::open_with_options so the production policy is applied; tests get the backend-aware local-FS defaults.

pub async fn export_clean_lance_datasets( &self, dest: &Path, ) -> Result<LanceArchiveExport>

Export clean, index-free Lance datasets into dest.

This rewrites the visible rows of each table instead of copying the dataset roots. The resulting manifests therefore contain no references to the source store’s _indices, while messages.vector and messages.embedding_model remain ordinary data columns and are preserved.

pub async fn import_clean_lance_datasets( &self, source: &Path, ) -> Result<LanceArchiveImport>

pub async fn all_session_message_counts(&self) -> Result<HashMap<String, usize>>

Per-session message count - the data-intrinsic freshness key for incremental pond copy. pond is append-only (merge is WhenMatched::DoNothing; no edits or deletes), so this count rises iff a session gained messages, catching growth a MAX(timestamp) key would miss when a new message shares the session’s latest timestamp. The count is source-authored and survives the copy unchanged, so it compares soundly across two stores with independent clocks (spec.md#session-durable-copy). Projects only the one column it counts; resolves the session_id array once per batch and allocates a key only on a session’s first row. Distinct from session_message_counts, which counts a supplied id list one query each; this counts every session in a single scan.

pub async fn all_session_part_counts(&self) -> Result<HashMap<String, usize>>

pub async fn plan_incremental_from(&self, source: &Store) -> Result<DeltaPlan>

Plan an incremental store-to-store copy into self from source, deciding per table whether each source session’s rows can be appended (the destination has none, so they cannot collide) or must be merged (the destination has some, source has more). Reads both id-sets plus per-session message and part counts. Parts have their own data-derived signal so a part added under an existing message routes through merge instead of relying on the closing verify to catch it (spec.md#session-movement-complete).

pub async fn copy_delta_from( &self, source: &Store, plan: &DeltaPlan, ) -> Result<LanceArchiveImport>

Copy the planned delta from source into self, streaming the source scan straight into the destination - no local staging copy. Each table picks its primitive per session from its TablePlan (spec.md#session-durable-copy): append the sessions whose rows are absent here (cannot collide; one commit per scan, bandwidth-bound) then merge the partially-present ones (WhenMatched::DoNothing dedups the rows already there). Append-only storage is what makes the append safe: a re-run re-plans from current destination state, so an interrupted-then-resumed copy never double-appends (landed rows are no longer absent).

pub async fn append_absent_rows( &self, source: &Store, table: Table, filter_column: &'static str, values: &[String], ) -> Result<usize>

Append source rows whose filter_column is in values. Absent rows can’t collide, so append is safe where the count-based plan would merge (spec.md#session-durable-copy). Drives the copy_bench append-vs-merge regression guard.

pub async fn upsert_sessions(&self, sessions: &[Session]) -> Result<()>

Flat write path. Per-row insert/match truth is not synthesized here - honest outcomes come from the pre-existence scan on Self::upsert_session_batch; the CLI sync and wire ingest paths use that, so these helpers only need to surface write failure.

pub async fn upsert_messages( &self, session: &Session, messages: &[MessageWrite<'_>], ) -> Result<()>

pub async fn upsert_parts(&self, parts: &[Part]) -> Result<()>

pub async fn get_session( &self, session_id: &str, ) -> Result<Option<SessionWithMessages>>

pub async fn session_ids(&self) -> Result<Vec<String>>

Every session id currently in the store, unsorted.

pub async fn child_sessions( &self, parent_session_id: &str, ) -> Result<Vec<Session>>

pub async fn session_last_message_ids(&self) -> Result<HashMap<String, String>>

session_id -> last durable message id for the sync freshness gate. Scans stored message data only, never Lance version history: Dataset::versions() is remote-manifest-bound on object stores, and a write timestamp can exist even when a non-atomic ingest did not commit the messages (spec.md#session-movement-complete).

Only emits a key when the session row is ALSO durable. upsert_session_batch commits messages+parts before the session row, so a partial flush can leave a session whose messages are stored but whose session row is not; keying on messages alone would report it fresh and orphan the missing row. Intersecting with the sessions id-set forces a re-ingest that heals it (spec.md#session-movement-complete).

pub async fn session_view( &self, session_id: &str, params: SessionViewParams<'_>, ) -> Result<GetLookup<SessionPage>>

Whole-session view for pond_get session scope (spec.md#protocol). Always the conversational view (search_text IS NOT NULL) with one-line part summaries - full part bodies are reached by message_id scope, not here. The page is the window selected by the anchors (after_message_id pages forward, before_message_id pages backward) or, with neither, session_from (start/end); it is bounded by limit and a byte budget, never cutting mid-message. before_remaining/after_remaining drive the bidirectional page markers.

pub async fn message_view( &self, message_id: &str, params: MessageViewParams, ) -> Result<GetLookup<MessagePage>>

Message-scope retrieval for pond_get message scope (spec.md#protocol): the target with its full parts (budget-bounded) plus context_before conversational siblings before and context_after after it. NotFound when no stored message carries message_id. Sibling parts are carried for summarizing; the target’s parts ride target_parts.

pub async fn scan_conversational_messages( &self, session_id: &str, ) -> Result<Vec<ConversationalRow>>

Conversational scan over one session: rows ordered by (timestamp, id), IsNotNull("search_text") pushed down at the read seam (spec.md#search-prefilter-pushdown).

pub async fn session_id_for_message( &self, message_id: &str, ) -> Result<Option<String>>

Locate the session id for a stored message. Cheap when only the routing hint is needed - callers that need the messages use scan_all_messages.

pub async fn row_counts(&self) -> Result<(usize, usize, usize)>

pub async fn collect_ids(&self, table: Table) -> Result<HashSet<String>>

The primary-key (id) set for table. Powers storage verification (pond copy --verify-only and copy’s closing check).

pub async fn id_diff_against( &self, table: Table, present: &HashSet<String>, ) -> Result<(usize, usize)>

Stream table’s id column and return (rows_scanned, rows whose id is absent from present). Streaming means the scanned side is never materialized into a set, so a verify holds only the present (other) side per table instead of both.

pub async fn dataset(&self, table: Table) -> Result<Arc<Dataset>>

A point-in-time Arc<Dataset> for table, for registering as a DataFusion LanceTableProvider in pond_sql_query. Goes through the handle’s freshness gate, so each query sees a current snapshot.

pub async fn prewarm(&self, cache_dir: &Path) -> Result<()>

Page the heavy search indices in from storage so the first user query after process start never eats the 175-442 s cold S3 load (spec.md#search). Vector via prewarm_index (loads the IVF_SQ partition storage). FTS is warmed with one synthetic query: Lance’s full FTS prewarm_index loads every token’s posting list, which resident-sets the whole inverted index (~1.7 GiB on the 2M-row corpus) and blows the server RAM budget - so we settle the term dictionary + a hot token instead and let real queries page their own postings. Best effort: a missing index (IVF_SQ below activation, or no FTS yet on an empty store) is logged, not fatal.

pub async fn ensure_rowmap(&self, cache_dir: &Path) -> Result<()>

Install the resident meta map covering the current messages version. Idempotent - a chain already at that version is kept. On a version bump it layers a delta segment (scanning only the new fragments), compacts the deltas locally once they pile up, and full-rebuilds the base only on a store compaction - all under a build flock so N local processes don’t rescan the store at once.

pub fn rowmap_snapshot(&self) -> Option<Arc<RowMetaSet>>

The currently-installed resident meta map, if any. pond sync reads it (via RowmapOracle) as the freshness oracle (max timestamp per session); None falls back to re-reading every source.

pub async fn export_write(&self, name: &str, bytes: &[u8]) -> Result<()>

Write a pond_sql_query export artifact.

pub async fn export_read(&self, name: &str) -> Result<Vec<u8>>

Read a pond_sql_query export artifact back.

pub fn export_local_path(&self, name: &str) -> Option<PathBuf>

Local filesystem path of an export artifact on file:// installs.

pub async fn adapter_names( &self, include_subagents: bool, ) -> Result<Vec<String>>

Distinct adapter names present in the corpus, sorted. Scans only the source_agent column of the small sessions table, so pond status gets its adapter count without touching the 2M-row messages table. include_subagents=false drops source_agent values containing / (e.g. claude-code/general-purpose).

pub async fn write_embeddings(&self, rows: &[EmbeddedMessage]) -> Result<()>

Write a batch of embeddings into messages: set vector and embedding_model on each row by (session_id, id) (spec.md#session-embed-from-canonical). The column update goes through the write seam and lands as a new manifest version (append-only).

pub fn pending_embedding_messages( &self, ) -> impl Stream<Item = Result<PendingMessage>> + '_

Stream the backlog of messages needing embedding: rows with search_text set whose vector is null (spec.md#session-embed-from-canonical).

pub fn pending_or_stale_messages( &self, ) -> impl Stream<Item = Result<PendingMessage>> + '_

Stream messages that are either never embedded or stale under the current model. pond optimize --force-embed feeds this to the same unconditional merge_update as the normal backlog; the filter makes that semantically equivalent to the conditional update in spec.md#session-embed-from-canonical.

pub async fn fts_search( &self, query: &str, limit: usize, filter: &Predicate, ) -> Result<Vec<SearchHit>>

BM25 full-text retriever over messages.search_text. With the row meta map loaded the scan is index-only (no data columns -> no TakeExec, no scattered GETs) and hits resolve through the map; otherwise it falls back to fts_search_keys so search works before prewarm.

pub async fn messages_version(&self) -> Result<u64>

Current messages dataset version - the key a RowMetaMap is built against (pond’s stable row ids keep a built map valid until this advances).

pub async fn collect_row_metas(&self) -> Result<Vec<RowMetaEntry>>

Scan the hydration columns with row ids into a Vec, the input to RowMetaMap::build. One large sequential scan (few big reads), unlike the scattered per-hit take it replaces; search_text dominates the bytes.

pub async fn searchable_in_scope(&self, filter: &Predicate) -> Result<usize>

Count of searchable messages (non-null search_text) inside the caller’s filter scope - the universe a search actually ran over. Powers the response’s absence honesty (spec.md#search): “no relevant hits” only means something relative to how many messages were searchable at all, and 0 tells the caller their filters excluded everything before retrieval even started.

pub async fn has_embeddings(&self) -> Result<bool>

Whether any messages row carries a vector (spec.md#search) - the signal that lets the vector arm run instead of degrading to fts. The single-active- model invariant (see MESSAGE_SCALAR_INDICES) means any non-null vector belongs to the current model.

pub async fn sample_embedded_model(&self) -> Result<Option<String>>

One embedded row’s model id, or None when nothing is embedded yet. A LIMIT 1 point read: the single-active-model invariant (see has_embeddings) means any embedded row’s model is representative, so a model swap is detectable by comparing this to the configured model - without the full-column stale_embedding_count scan that ran every sync.

pub async fn vector_search( &self, query: &[f32], limit: usize, filter: &Predicate, search: Option<&SearchConfig>, ) -> Result<Vec<SearchHit>>

Vector kNN retriever over messages.vector, prefiltered by the caller’s scalar predicate alone (spec.md#search-prefilter-pushdown) - see embedded_scope for why pond does NOT add vector IS NOT NULL. nprobes falls back to DEFAULT_NPROBES when [search] leaves it unset, so a default install never inherits Lance’s unbounded “probe every partition” behavior on a remote store. No refine (see apply_vector_search_knobs). Index-only + map resolve when loaded, else key projection - see fts_search.

pub async fn explain_vector_plan( &self, query: &[f32], limit: usize, filter: &Predicate, search: Option<&SearchConfig>, ) -> Result<String>

The DataFusion plan string for a filtered vector scan - the search-prefilter-pushdown regression guard reads it.

pub async fn explain_fts_plan( &self, query: &str, limit: usize, filter: &Predicate, ) -> Result<String>

pub async fn message_metas_by_rowids( &self, rowids: &[u64], ) -> Result<Vec<MessageMeta>>

Hydrate search hits by stable row id (spec.md#search). Resolves each rowid from the resident meta map in memory (no object-store round-trip - Lance caches index/metadata but never data column values, so a take_rows re-reads search_text from storage every query). Rowids the map lacks (appended since it was built, or no map loaded) fall back to a single take_rows batch. The caller indexes the result by key, so order is irrelevant.

pub async fn message_metas_by_keys( &self, keys: &[MessageKey], ) -> Result<Vec<MessageMeta>>

Hydrate search hits: fetch message metadata for (session_id, message_id) keys.

pub async fn session_message_counts( &self, session_ids: &[String], ) -> Result<BTreeMap<String, usize>>

Total message count per session, for search session summaries. One session_id IN (...) scan projecting only session_id, aggregated in pond, instead of N concurrent count_rows(session_id = X) round-trips against messages_session_id_btree. Same wire shape for any backend, but one S3 operation instead of N on remote stores. Sessions with zero matching messages are present in the map with count 0 so the caller can distinguish “filter excluded everything” from “session missing from the response.”

pub async fn unindexed_message_backlog(&self) -> Result<usize>

Rows appended to messages since the FTS index was last optimized. A missing index reports the whole table; the query is manifest-only.

pub async fn unindexed_vector_backlog(&self) -> Result<usize>

Rows added or rewritten in messages since the IVF_SQ vector index was last optimized. Below VECTOR_INDEX_ACTIVATION_ROWS no index exists yet, so the caller must read embedding_progress too and distinguish “index not built yet” from “index trails data”.

pub async fn embedding_progress(&self) -> Result<EmbeddingProgress>

Embedding coverage: how many messages rows carry a vector and how many are still eligible. Drives the pond status embeddings line and the pond optimize progress bar’s known total.

pub async fn embed_backlog_count(&self) -> Result<usize>

Messages eligible but not yet embedded (search_text present, embedding_model null) - the exact set crate::embed::EmbedWorker processes. Read straight from the dataset so it is correct right after ingest, unlike the FTS num_docs embedding_progress shows (which lags until the index is rebuilt - the embed stage runs before that).

pub async fn stale_embedding_count(&self) -> Result<usize>

Count rows whose embedding_model is not the currently configured model AND whose vector is still populated - the signal pond optimize uses to detect a model swap and require --force-embed.

pub async fn optimize_indices( &self, progress: Option<OptimizeProgressFn>, maintenance: &MaintenancePolicy, ) -> Result<OptimizeOutcome>

Run the per-table maintenance cycle (compact + indices) across every table, never short-circuiting. spec.md#lance-index-maintenance: indices and compaction commit independently, so a hot writer that starves compaction on one table does not abort the index work the operator asked for on other tables (or even on the same table).

pub async fn build_indices_only( &self, progress: Option<OptimizeProgressFn>, ) -> Result<OptimizeOutcome>

Fold trailing fragments into existing indices across every table, without running compaction. Used by pond optimize’s tail so newly written vectors land in the FTS / IVF_SQ / btree / bitmap indices without paying the compaction retry budget while embed itself may still be writing in a sibling process.

pub async fn cleanup_old_versions(&self, older_than: Duration) -> Result<()>

Reclaim superseded data/index files across every indexed table (Lance cleanup_old_versions), without compaction. pond optimize --rebuild runs this after the rebuild so the index segments it just replaced are dropped immediately. The retention floor still protects versions a live reader may have pinned (spec.md#concurrency).

pub async fn rebuild_indices( &self, intent_name: Option<&str>, progress: Option<OptimizeProgressFn>, ) -> Result<()>

pub async fn drop_index_by_name(&self, name: &str) -> Result<()>

Drop a named index from whichever table owns it. Used by pond optimize --drop-index <name> to clean up orphaned indices (e.g. after renaming an intent whose on-disk name no longer matches the policy). Finds the owning table via parallel load_indices lookups, then drops on just that table - so real I/O errors surface with the right context instead of being hidden behind “no such index” from the wrong table.

pub async fn index_status(&self) -> Result<Vec<IndexStatus>>

pub async fn drop_vector_index(&self) -> Result<()>

Drop the IVF_SQ index on messages.vector. Used by pond optimize --force-embed before re-bootstrapping under a different model. Silent when the index does not exist.

pub async fn table_sizes(&self) -> Result<TableSizes>

On-disk byte totals per dataset, sized through Lance’s object store (spec.md#lance-chokepoints-storage) so pond status works on any backend.

pub async fn initialized(&self) -> Result<bool>

pub async fn parts_for_messages( &self, session_id: &str, message_ids: &[String], ) -> Result<BTreeMap<(String, String), Vec<Part>>>

Every part of these messages, full fidelity (file blobs included). The canonical read primitive - restore/export, verbatim mode, and the message-mode target all need the complete set.

pub async fn summary_parts_for_messages( &self, session_id: &str, message_ids: &[String], ) -> Result<BTreeMap<(String, String), Vec<Part>>>

Only the parts that yield a [PartSummary] (SUMMARY_PART_TYPES), skipping text/reasoning (and their blobs) that would summarize to nothing. For the summary-only reads (conversational/complete session views, search hits) - it never feeds restore/export.

Trait Implementations

impl Debug for Store

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.