Struct EmbedWorker 

pub struct EmbedWorker<'a, B: Embedder> { /* private fields */ }

Fills messages.vector / messages.embedding_model for the backlog of un-embedded messages. Reads messages.search_text directly, batches it through the backend one vector each, and writes each batch back to messages by primary key.

Implementations

impl<'a, B: Embedder> EmbedWorker<'a, B>

pub fn new(store: &'a Store, backend: &'a B) -> Self

Build a worker over store’s un-embedded backlog. A backend whose vectors are the wrong width is rejected at the write boundary (embedding_update_batch), so there is nothing to validate here.

pub fn with_cancel(self, flag: Arc<AtomicBool>) -> Self

Honour flag as a cooperative cancellation signal. The pull loop checks it before each new stream message; once set, the worker drains the current window (committing the embedded slice) and returns with EmbedSummary { cancelled: true, .. }. pond embed wires this to a Ctrl-C handler so an interrupted run doesn’t lose its in-memory window.

pub fn with_sort_window(self, window: usize) -> Self

Override the length-sort window (default DEFAULT_SORT_WINDOW). The benchmark harness sweeps this to size the padding-waste vs. throughput trade-off; a window of DEFAULT_BATCH_SIZE disables sorting.

pub fn with_progress( self, callback: impl Fn(BatchProgress) + Send + Sync + 'static, ) -> Self

Register a per-batch progress callback. Called once after each flush() with the messages in the just-finished batch and the running totals. pond embed uses this to drive an indicatif progress bar.

pub fn with_limit(self, limit: usize) -> Self

Cap the run at limit messages (default: no cap). The benchmark harness uses this to embed a fixed, comparable slice of a corpus.

pub fn include_stale(self) -> Self

pub async fn run(&self) -> Result<EmbedSummary>

Embed every message whose vector is still null. Idempotent: a re-run over an already-embedded corpus finds an empty backlog and is a no-op.

Messages are pulled from a streaming scan, so peak memory is one stream page plus the staged batch - not the whole corpus.