Struct Store 

pub struct Store { /* private fields */ }

Implementations

impl Store

pub async fn open(location: &Url) -> Result<Self>

Open against a local filesystem URL or a remote one for which the caller has no extra options to pass (env vars suffice). CLI verbs that load [storage] from config should call Store::open_with_options instead so the same options flow into every dataset open and write.

pub async fn open_with_options( location: &Url, storage_options: HashMap<String, String>, caps: RuntimeCaps, ) -> Result<Self>

Open with object-store options (S3 creds, region, endpoint, …) threaded through Lance verbatim. Keys are the standard object_store config names; pond does not parse them. Empty options + default caps is equivalent to Store::open. Cache caps come from the [runtime] config block via crate::substrate::RuntimeCaps.

pub async fn open_local(path: impl AsRef<Path>) -> Result<Self>

Convenience for tests and CLI verbs holding a &Path: wraps the path in a file://... URL via config::url_for_path before opening. Routes through Store::open_with_options so the production policy is applied; tests get the backend-aware local-FS defaults.

pub async fn export_clean_lance_datasets( &self, dest: &Path, ) -> Result<LanceArchiveExport>

Export clean, index-free Lance datasets into dest.

This rewrites the visible rows of each table instead of copying the dataset roots. The resulting manifests therefore contain no references to the source store’s _indices, while messages.vector and messages.embedding_model remain ordinary data columns and are preserved.

pub async fn import_clean_lance_datasets( &self, source: &Path, ) -> Result<LanceArchiveImport>

pub async fn upsert_sessions( &self, sessions: &[Session], ) -> Result<Vec<UpsertStatus>>

pub async fn upsert_messages( &self, session: &Session, messages: &[MessageWrite<'_>], ) -> Result<Vec<UpsertStatus>>

pub async fn upsert_parts(&self, parts: &[Part]) -> Result<Vec<UpsertStatus>>

pub async fn get_session( &self, session_id: &str, ) -> Result<Option<SessionWithMessages>>

pub async fn session_ids(&self) -> Result<Vec<String>>

Every session id currently in the store, unsorted.

pub async fn child_sessions( &self, parent_session_id: &str, ) -> Result<Vec<Session>>

pub async fn session_last_ingested_at( &self, ) -> Result<HashMap<String, DateTime<Utc>>>

session_id -> wall-clock time of the Lance manifest version that last wrote the row for the per-session staleness skip (spec.md#adapter-integrity-event-ordering). Reads Lance’s _row_last_updated_at_version system column (available because pond enables stable row ids per spec.md#lance-table-creation-stable-row-ids) and joins it against Dataset::versions() for commit timestamps.

pub async fn session_view( &self, session_id: &str, params: SessionViewParams<'_>, ) -> Result<GetLookup<SessionPage>>

Whole-session view for pond_get session mode (spec.md#protocol). Conversational filters to search_text IS NOT NULL; Complete and Verbatim scan every message. Every mode attaches compact part summaries; Verbatim additionally inlines full parts. after_id is an exclusive lower bound (a message id); the page is bounded by limit and a byte budget and never cuts mid-message.

pub async fn message_view( &self, message_id: &str, params: MessageViewParams<'_>, ) -> Result<GetLookup<MessagePage>>

Message-scope retrieval for pond_get message mode (spec.md#protocol): the target with its full parts (paginated by after_id over part ordinals, then budget) plus up to 2*context_depth siblings around it. None when no stored message carries message_id. Sibling parts are carried for summarizing; the target’s parts ride target_parts.

pub async fn scan_conversational_messages( &self, session_id: &str, ) -> Result<Vec<ConversationalRow>>

Conversational scan over one session: rows ordered by (timestamp, id), IsNotNull("search_text") pushed down at the read seam (spec.md#search-prefilter-pushdown).

pub async fn session_id_for_message( &self, message_id: &str, ) -> Result<Option<String>>

Locate the session id for a stored message. Cheap when only the routing hint is needed - callers that need the messages use scan_all_messages.

pub async fn row_counts(&self) -> Result<(usize, usize, usize)>

pub async fn corpus_stats(&self, include_subagents: bool) -> Result<CorpusStats>

Compute the per-adapter / per-project rollup that drives pond status. One scan over messages projecting the three columns the rollup keys on (source_agent, project, session_id), aggregated in-memory. Bounded by the cross product of adapters and projects, which stays small on real corpora.

pub async fn write_embeddings(&self, rows: &[EmbeddedMessage]) -> Result<()>

Write a batch of embeddings into messages: set vector and embedding_model on each row by (session_id, id) (spec.md#session-embed-from-canonical). The column update goes through the write seam and lands as a new manifest version (append-only).

pub fn pending_embedding_messages( &self, ) -> impl Stream<Item = Result<PendingMessage>> + '_

Stream the backlog of messages needing embedding: rows with search_text set whose vector is null (spec.md#session-embed-from-canonical).

pub fn pending_or_stale_messages( &self, ) -> impl Stream<Item = Result<PendingMessage>> + '_

Stream messages that are either never embedded or stale under the current model. pond embed --force feeds this to the same unconditional merge_update as the normal backlog; the filter makes that semantically equivalent to the conditional update in spec.md#session-embed-from-canonical.

pub async fn fts_search( &self, query: &str, limit: usize, filter: &Predicate, ) -> Result<Vec<(MessageKey, f32)>>

BM25 full-text retriever over messages.search_text.

pub async fn has_embeddings(&self) -> Result<bool>

Whether any messages row carries a vector (spec.md#search) - the signal that flips search from FTS-only to hybrid. The single-active- model invariant (see MESSAGE_SCALAR_INDICES) means any non-null vector belongs to the current model.

pub async fn vector_search( &self, query: &[f32], limit: usize, filter: &Predicate, search: Option<&SearchConfig>, ) -> Result<Vec<(MessageKey, f32)>>

Vector kNN retriever over messages.vector, prefiltered by the caller’s scalar predicate (spec.md#search-prefilter-pushdown). Combines the caller’s filter with vector IS NOT NULL to exclude un-embedded rows from the scan; the brute-force kNN path requires this (the IVF_PQ path would skip them anyway). The single-active-model invariant lets pond drop the per-row model filter: every non-null vector belongs to the current model.

pub async fn explain_vector_plan( &self, query: &[f32], limit: usize, filter: &Predicate, search: Option<&SearchConfig>, ) -> Result<String>

The DataFusion plan string for a filtered vector scan - the search-prefilter-pushdown regression guard reads it.

pub async fn explain_fts_plan( &self, query: &str, limit: usize, filter: &Predicate, ) -> Result<String>

pub async fn message_metas_by_keys( &self, keys: &[MessageKey], ) -> Result<Vec<MessageMeta>>

Hydrate search hits: fetch message metadata for (session_id, message_id) keys.

pub async fn session_message_counts( &self, session_ids: &[String], ) -> Result<BTreeMap<String, usize>>

Total message count per session, for search session summaries.

pub async fn unindexed_message_backlog(&self) -> Result<usize>

Rows appended to messages since the FTS index was last optimized. A missing index reports the whole table; the query is manifest-only.

pub async fn unindexed_vector_backlog(&self) -> Result<usize>

Rows added or rewritten in messages since the IVF_PQ vector index was last optimized. Below VECTOR_INDEX_ACTIVATION_ROWS no index exists yet, so the caller must read embedding_progress too and distinguish “index not built yet” from “index trails data”.

pub async fn embedding_progress(&self) -> Result<EmbeddingProgress>

Embedding coverage: how many messages rows carry a vector and how many are still eligible. Drives the pond status embeddings line and the pond embed progress bar’s known total. embedded reads the vector IS NOT NULL count directly - the single-active-model invariant (see MESSAGE_SCALAR_INDICES) means there is no need to scope by the embedding_model column.

pub async fn stale_embedding_count(&self) -> Result<usize>

Count rows whose embedding_model is not the currently configured model AND whose vector is still populated - the signal pond embed uses to detect a model swap and require --force.

pub async fn optimize_indices( &self, progress: Option<OptimizeProgressFn>, cleanup: Option<CleanupConfig>, ) -> Result<OptimizeOutcome>

Run the per-table maintenance cycle (compact + indices) across every table, never short-circuiting. spec.md#lance-index-maintenance: indices and compaction commit independently, so a hot writer that starves compaction on one table does not abort the index work the operator asked for on other tables (or even on the same table).

pub async fn build_indices_only( &self, progress: Option<OptimizeProgressFn>, ) -> Result<OptimizeOutcome>

Fold trailing fragments into existing indices across every table, without running compaction. Used by pond embed’s tail so newly written vectors land in the FTS / IVF_PQ / btree / bitmap indices without paying the compaction retry budget while embed itself may still be writing in a sibling process.

pub async fn rebuild_indices(&self, intent_name: Option<&str>) -> Result<()>

pub async fn index_status(&self) -> Result<Vec<IndexStatus>>

pub async fn drop_vector_index(&self) -> Result<()>

Drop the IVF_PQ index on messages.vector. Used by pond embed --force before re-bootstrapping under a different model. Silent when the index does not exist.

pub async fn table_sizes(&self) -> Result<TableSizes>

On-disk byte totals per dataset, sized through Lance’s object store (spec.md#lance-chokepoints-storage) so pond status works on any backend.

pub async fn text_script_histogram( &self, max_messages: usize, ) -> Result<Vec<(String, usize)>>

Histogram of Unicode script classes in messages.search_text, computed from a sample of up to max_messages non-null rows. Returned classes are sorted descending by character count. Lets pond status tell an agent whether the corpus is monolingual or mixed - the agent then knows whether bilingual querying is worth attempting (cross-lingual recall is a caller-layer concern; pond does not translate internally).

pub async fn message_vector_by_id( &self, message_id: &str, ) -> Result<Option<Vec<f32>>>

Fetch the stored vector for one message, for similar_to-style retrieval. Returns Ok(None) when the message exists but is not yet embedded, or when no message has that id. Vectors are stored Float16 (embedding_vector_type); the search path takes &[f32], so we widen here. Cheap rare op - one predicate scan, no index.

pub async fn parts_for_messages( &self, session_id: &str, message_ids: &[String], ) -> Result<BTreeMap<(String, String), Vec<Part>>>

Every part of these messages, full fidelity (file blobs included). The canonical read primitive - restore/export, verbatim mode, and the message-mode target all need the complete set.

pub async fn summary_parts_for_messages( &self, session_id: &str, message_ids: &[String], ) -> Result<BTreeMap<(String, String), Vec<Part>>>

Only the parts that yield a [PartSummary] (SUMMARY_PART_TYPES), skipping text/reasoning (and their blobs) that would summarize to nothing. For the summary-only reads (conversational/complete session views, search hits) - it never feeds restore/export.

Trait Implementations

impl Debug for Store

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.