Struct LazyEmbedder 

pub struct LazyEmbedder { /* private fields */ }

Lazy holder for an Embedder with idle eviction. The model isn’t loaded until the first hybrid/vector call asks for it - idle pond mcp / pond serve processes pay nothing while no vector queries land. After idle_threshold of inactivity the cached backend is dropped on the next get call; under macOS phys_footprint the drop reclaims ~365-585 MiB cleanly (the post-drop floor is ~107 MiB regardless of backend). Reload cost is one synchronous model-load (300-500 ms), absorbed inside the human-paced gap between MCP queries.

Implementations

impl LazyEmbedder

pub fn candle() -> Self

candle XLM-RoBERTa FP16 (Metal on macOS / CUDA with --features cuda / CPU otherwise). The pond default for every entry point.

pub fn with_loader( loader: Arc<dyn Fn() -> Result<Arc<dyn Embedder>> + Send + Sync>, ) -> Self

Build a LazyEmbedder from an explicit loader. Used by the bench harness to override the idle threshold; production callers use Self::candle.

pub fn with_idle_threshold(self, threshold: Duration) -> Self

Override the idle-eviction threshold. Pass Duration::MAX to disable eviction entirely - useful in benches that want a stable steady-state.

pub fn from_loaded(backend: Arc<dyn Embedder>) -> Self

Pre-seed with an already-constructed backend. Used by integration tests that want to inject a fake Embedder without paying the real model-load cost. Eviction is disabled so the test fake survives the whole test even if a test stalls.

pub async fn get(&self) -> Result<Arc<dyn Embedder>>

Load (on first call or after eviction) or return the cached handle. The candle load is synchronous and blocking, so it runs on spawn_blocking; the async caller sees a clean await point.