Struct InferenceEngine 

pub struct InferenceEngine {
    pub config: InferenceConfig,
    pub adaptive_router: AdaptiveRouter,
    pub outcome_tracker: Arc<RwLock<OutcomeTracker>>,
    pub registry: ModelRegistry,
    pub router: ModelRouter,
    /* private fields */
}

Fields

config: InferenceConfig

adaptive_router: AdaptiveRouter

Adaptive router with three-phase selection.

outcome_tracker: Arc<RwLock<OutcomeTracker>>

Outcome tracker for learning from results.

registry: ModelRegistry

router: ModelRouter

Implementations

impl InferenceEngine

pub fn local_model_retention(&self, model_id: &str) -> BackendRetention

Actual post-dispatch retention observed by worker/process owners. Success alone is insufficient: zero-cache loads remain transient.

pub fn local_model_preflight( &self, model_id: &str, context_tokens: usize, ) -> Result<LocalLoadPreflight, InferenceError>

Evaluate one local model without downloading or loading it.

pub fn active_local_resource_policy(&self) -> ResourcePolicyLoadEvidence

The policy currently enforced by local-model admission, plus the load source/warning captured when this engine initialized it.

Read-side model surfaces use this accessor rather than reopening the policy file, so a policy applied to the running engine is one atomic source of truth for preflight, fit annotations, and recommendations.

pub fn apply_local_resource_policy(&self, policy: ResourcePolicy)

Update the in-memory admission/cache ceiling after persistence succeeds. Idle entries are reclaimed by each cache’s next sweep/access; active inference is never killed by a policy decrease.

pub fn begin_local_model_maintenance( &self, model_id: &str, ) -> Result<LocalModelMaintenanceGuard, ModelMaintenanceError>

pub async fn prepare_local_model_removal( &self, model_id: &str, ) -> Result<LocalModelMaintenanceGuard, ModelMaintenanceError>

pub fn evict_local_model_if_idle(&self, model_id: &str) -> bool

Targeted cache eviction for safe model removal. Callers must hold the per-model maintenance guard while invoking this and checking any worker/cross-process leases.

pub fn set_spend_limits(&self, limits: Option<SpendLimits>)

Install (or clear) spend limits (I4). per_request_usd also arms the mid-stream guard on streaming calls: the stream is cancelled with a terminal StopReason("spend_limit: ...") the moment the estimated running cost (prompt + streamed output) crosses the budget.

pub fn new(config: InferenceConfig) -> Self

pub async fn init_key_pool(&self)

Initialize key pool: register keys from all remote models and load persisted stats. Call this after construction (requires async).

pub async fn reset_local_kv_cache(&self, model_id: &str)

Clear the in-process KV / prefix cache of a loaded local model.

Prefix reuse (begin_prompt) is a per-conversation optimization: it reuses the KV state of a shared token prefix across calls. When one engine is driven through a sequence of independent prompts (e.g. a benchmark’s task suite), that reuse leaks decode state between unrelated conversations — and reusing cached KV instead of a fresh prefill introduces tiny numerical drift that can flip a greedy (temperature-0) token, making multi-step runs non-reproducible. Calling this between independent runs restores a clean slate. No-op for remote models or a backend that isn’t currently loaded.

pub async fn warm_up<S: AsRef<str>>( &self, _schema_ids: &[S], ) -> Vec<Result<(), InferenceError>>

No-op on non-macOS — MLX doesn’t run here.

pub async fn evict_idle_vllm_servers(&self) -> usize

Stop idle supervised vllm-mlx servers. Driven by the same idle loop that evicts in-process backends; returns the number stopped.

pub fn evict_idle_backends(&self) -> (usize, u64)

No-op on non-macOS — there are no accumulating backend caches.

pub fn unified_registry(&self) -> Arc<UnifiedRegistry>

The registry generation this caller should read: the one pinned for the operation in progress (see Self::with_pinned_registry), else the current one. Cheap: a read lock and an Arc clone. Hold the returned Arc — not the lock — across an .await.

pub fn unified_registry_mut(&mut self) -> &mut UnifiedRegistry

Exclusive access for &mut self callers: no other reader can hold the engine, so this edits in place, cloning only if a reader still holds an older generation’s Arc.

pub fn swap_signed_catalog(&self, catalog: VerifiedCatalog) -> usize

Load a newly verified signed catalog into the running registry: its rows replace the previous signed catalog’s, so a row the publisher dropped is withdrawn, and the same merge rules as startup apply — a compiled builtin id is never overwritten. Only a catalog::VerifiedCatalog is accepted, so an unverified row cannot reach a running registry. Rows kept that way are persisted with the catalog they came from, so a restart keeps them too. Returns how many signed rows the registry now carries.

pub async fn route_adaptive(&self, prompt: &str) -> AdaptiveRoutingDecision

Route a prompt using the adaptive router (new). Returns full decision context.

pub async fn route_adaptive_with_intent( &self, prompt: &str, intent: Option<IntentHint>, ) -> AdaptiveRoutingDecision

Like route_adaptive but honors a caller IntentHint — notably exclude_models for adversarial-reviewer separation: “route me any capable model that is NOT the one that just did the work” (car#358). An excluded id is never chosen while any non-excluded capable model exists — including via the preferred-model override (skipped when it names an excluded model) and the cold-start fallbacks. The exclusion is soft: if excluding leaves nothing routable, an excluded model may still be returned as a last resort (a same-model review beats no review).

pub fn route(&self, prompt: &str) -> RoutingDecision

Route a prompt to the best model without executing (legacy compat).

pub fn estimated_tokens( &self, req: &GenerateRequest, model_id: Option<&str>, ) -> (usize, usize, bool)

Estimate token count for a request against a specific model’s context window. Returns (estimated_input_tokens, context_window_tokens, fits).

Multimodal content blocks (image/video/audio, in images or in messages history) contribute provider-calibrated estimates via media_tokens — a minute of video is ~15.8K input tokens at Gemini’s documented rate, not zero — and the multi-turn messages history’s text is counted too (chars/4), not just its media. This feeds the adaptive router’s window-fit / needs_compaction signal.

pub fn model_context_window(&self, model_id: &str) -> usize

The model’s context window in tokens, or 0 if the id is unknown (unregistered). Public so a multi-turn driver (e.g. the assistant loop) can bound its running message history to the window before it overflows — an overflowed history pushes the model to its context limit and can truncate the original task provider-side.

pub async fn generate_tracked( &self, req: GenerateRequest, ) -> Result<InferenceResult, InferenceError>

Generate text with full tracking (tool_calls, usage, trace_id, latency, TTFT), plus Qwen3 hybrid-thinking recovery.

Qwen3 (and other hybrid-thinking models) default to reasoning ON. With a small max_tokens budget the model can spend the entire budget inside an unclosed <think> block, so the strip pass returns empty text — infer(prompt, model, 16) then silently yields “” while a non-thinking model answers fine (car-releases#60, #62).

When the caller left thinking on Auto (didn’t explicitly opt into reasoning) and nothing usable came back, retry once with reasoning suppressed so the caller gets a direct answer — matching the CLI’s --thinking off default, but for every FFI/daemon path. Either way, record why via stop_reason so an empty result is never silent.

pub async fn generate_tracked_with_retry_observer( &self, req: GenerateRequest, retry_observer: &mut (dyn FnMut(InferenceRetryProgress) + Send), ) -> Result<InferenceResult, InferenceError>

Generate text while reporting retries that happen inside a provider request. The ordinary Self::generate_tracked path is identical but discards these progress notifications.

pub async fn generate_tracked_with_observers( &self, req: GenerateRequest, retry_observer: &mut (dyn FnMut(InferenceRetryProgress) + Send), exclusion_observer: &mut (dyn FnMut(RouterExclusion) + Send), ) -> Result<InferenceResult, InferenceError>

Generate text while reporting both provider retries and lanes that the adaptive router removed before dispatch. Router exclusions carry only bounded operational metadata and never provider response text.

pub async fn generate_tracked_stream( &self, req: GenerateRequest, ) -> Result<TrackedStream, InferenceError>

Stream a generation and record an outcome when it finishes.

Wraps generate_stream_raw with a forwarding “tap” task: it accumulates the event stream (via stream::StreamAccumulator), forwards every event to the caller unchanged, and on completion books a success/failure against the model’s profile — the same outcome telemetry the non-streaming generate_tracked path records. Without this, every streamed inference (voice/realtime, the daemon’s infer stream) was invisible to model-health scoring. Cancellation propagates: if the caller drops the returned receiver, the tap stops forwarding, drops the producer receiver, and the backend task observes the closed channel.

Returns a TrackedStream carrying the resolved model + this call’s trace_id alongside the event receiver. The trace_id is known up front (minted by record_start before the first token), so a caller can score the finished turn against the same trace the tap will resolve — the streaming counterpart of the non-streaming path’s InferenceResult {trace_id, model_used}, which the conversation-outcome signal needs.

The events receiver yields StreamEvent variants (TextDelta, ToolCallStart, ToolCallDelta, Usage, StopReason, ProviderOutputItem, Error, Done); use a stream::StreamAccumulator to collect them into a final result. Local backends (MLX, Candle) emit true incremental TextDeltas per token, enabling token-by-token UI, overlapping TTS, and early cancellation. The channel buffers 64 events so burst tokens don’t block generation.

Example: voice app integration

let mut handle = engine.generate_tracked_stream(req).await?;
let mut text_buf = String::new();
while let Some(event) = handle.events.recv().await {
    match event {
        StreamEvent::TextDelta(delta) => {
            text_buf.push_str(&delta);
            // Feed text_buf to TTS when a sentence boundary is reached
        }
        StreamEvent::Done { text, .. } => break,
        _ => {}
    }
}
// handle.trace_id / handle.model_used identify the turn for scoring.

pub async fn route_context_snapshot( &self, prompt: &str, workload: RoutingWorkload, has_tools: bool, has_vision: bool, ) -> AdaptiveRoutingDecision

Route a prompt using the adaptive router without executing inference.

pub async fn generate( &self, req: GenerateRequest, ) -> Result<String, InferenceError>

Generate text from a prompt (legacy API, no outcome tracking). When req.model is None, uses intelligent routing based on prompt complexity.

pub async fn tokenize( &self, model: &str, text: &str, ) -> Result<Vec<u32>, InferenceError>

Encode text via the named model’s tokenizer. Returns raw token IDs without any chat-template wrapping or BOS-prepending — pair with Self::detokenize for the round-trip property detokenize(model, tokenize(model, s)) == s for any UTF-8 s.

Only local models have a tokenizer the runtime can call directly (Candle/GGUF on Linux/Windows, MLX on Apple Silicon). For remote models the call returns InferenceError::UnsupportedMode — provider tokenizer endpoints vary too widely to be portable here, and bundling tiktoken-style tables would lock the registry to a fixed set of providers.

pub async fn detokenize( &self, model: &str, tokens: &[u32], ) -> Result<String, InferenceError>

Inverse of Self::tokenize: decode token IDs back to text.

pub async fn embed( &self, req: EmbedRequest, ) -> Result<Vec<Vec<f32>>, InferenceError>

Generate embeddings for text using the dedicated embedding model. On Apple Silicon, uses the native MLX backend; on other platforms, uses Candle.

pub async fn rerank( &self, req: RerankRequest, ) -> Result<RerankResult, InferenceError>

Rerank candidate documents against a query using a cross-encoder reranker model (Qwen3-Reranker family). Returns documents sorted by descending relevance.

Scoring

Qwen3-Reranker is a Qwen3 base LM fine-tuned so that the first assistant token is "yes" or "no" given the templated <Instruct>/<Query>/<Document> user turn. We run a short greedy decode (≤ 3 tokens, so a leading space, BOS artifact, or the occasional newline don’t break us) and score yes → 1.0, no → 0.0, anything else → 0.5 with a warning.

This is a binary score. The soft probability softmax(logit_yes, logit_no) would give finer ordering, and the logits it needs are already reachable: SwiftLmBackend implements TextDecoder::forward, which returns the last position’s full logits from one pass. What is missing is only a scoring path that reads them instead of going through generate — the same path a local System-1 decision (read option logits, no sampling) needs; see docs/proposals/system-one-decisions.md. Binary scores still produce a correct partial ordering, just with coarser tiebreaks within the {yes} or {no} groups.

Prompt template

We emit the upstream Qwen3-Reranker chat template verbatim: a dedicated system prompt fixing the yes/no answer space, then the user turn with <Instruct>/<Query>/<Document>, then the assistant prefix with a closed empty <think> block to suppress thinking (reranker is not a reasoner — it’s a classifier). Deviating from this template produces sharply degraded yes/no distributions.

pub async fn ground( &self, req: GroundRequest, ) -> Result<GroundResult, InferenceError>

Dedicated endpoint for structured visual grounding.

Runs a VL generate call under the hood and parses Qwen2.5-VL’s inline <|object_ref_*|>...<|box_*|>(x1,y1),(x2,y2) spans into typed BoundingBoxes. Distinct from the generic InferenceEngine::generate + InferenceResult.bounding_boxes path so callers can express “I want boxes” as a first-class intent — which also lets the router prefer models that declare the Grounding capability.

pub async fn classify( &self, req: ClassifyRequest, ) -> Result<Vec<ClassifyResult>, InferenceError>

Classify text against candidate labels. When req.model is None, routes to the smallest available model.

pub async fn option_probabilities( &self, model: &str, prompt: &str, options: &[String], ) -> Result<OptionProbabilities, InferenceError>

A local System One decision: the model’s probability for each of options as the start of its answer to prompt, read from logits with nothing sampled.

Experimental (docs/proposals/system-one-decisions.md, Experiment 5): Rust-only, no binding, used by car models classify-bench to ask whether a small local model read this way can route the voice turn.

Each option counts as its lowercase and capitalized spellings (a chat model often answers “Email”). When no two options share a first token across those spellings, one forward pass scores every option by its first token (method: "first_token"). Otherwise each spelling is scored as a whole sequence ending in the end-of-turn token ("sequence"), which favors shorter labels; the result says which ran. mass is the probability the options received before renormalizing: a low value means the model wanted to say something else, and the pick should not be read as confident. Qwen-family only, because the prompt is rendered with Qwen’s chat template.

pub async fn transcribe( &self, req: TranscribeRequest, ) -> Result<TranscribeResult, InferenceError>

Transcribe an audio file using the best available STT model.

pub async fn synthesize( &self, req: SynthesizeRequest, ) -> Result<SynthesizeResult, InferenceError>

Synthesize speech using the best available TTS model.

pub async fn generate_image( &self, req: GenerateImageRequest, ) -> Result<GenerateImageResult, InferenceError>

Generate an image through the managed mflux runtime on Apple Silicon.

pub async fn generate_image_batch( &self, req: GenerateImageRequest, ) -> Result<Vec<GenerateImageResult>, InferenceError>

Generate one or more variants in a single call.

Returns req.variant_count results (defaulting to 1). The current MLX Flux backend doesn’t support native batching, so this loops over generate_image with the seed advanced per variant for visual diversity. A future hosted backend (gpt-image-2, Replicate) can short-circuit this with one network call producing N coherent images.

Per-variant errors abort the batch — there’s no partial- success semantics today. Callers needing more lenient behaviour should call generate_image directly in their own loop.

Closes #110.

pub async fn generate_video( &self, req: GenerateVideoRequest, ) -> Result<GenerateVideoResult, InferenceError>

Generate a video using the best available local MLX video model.

pub fn response_format_rejection_reason( &self, model: &str, rf: &ResponseFormat, ) -> Option<String>

List all known models and their status (new registry). Why model cannot honor a response_format, or None when it can (or is unknown here). Asks the SAME protocol handler the remote path consults (ProtocolHandler::supports_response_format) so a CLI can warn before a run rather than discover the UnsupportedMode on the repair turn. The Parslee gateway rejects every format separately in execute_request, so it is named here too.

pub fn list_models_unified(&self) -> Vec<ModelInfo>

The unified catalog, annotated for the machine this engine runs on under its active local-model resource policy — the same policy models.preflight admits against, kept in step with the persisted one by apply_local_resource_policy.

pub fn model_fit(&self, schema: &ModelSchema) -> ModelFit

The fit annotation for one schema on this machine under the active policy — list_models_unified’s verdict for a row built elsewhere (models.search), so every catalog surface publishes the same one.

pub fn list_models_unified_for( &self, hardware: &HardwareInfo, policy: &ResourcePolicy, ) -> Vec<ModelInfo>

Self::list_models_unified against explicit hardware and policy. Every row is returned in registry order with every existing field unchanged; the fit annotation is computed per call and never stored.

pub fn list_models_unified_for_explicit_status(&self) -> Vec<ModelInfo>

Self::list_models_unified for a surface a person explicitly asked for a status from (car models list), resolving credential-backed availability authoritatively — environment or OS keychain.

The ordinary path refreshes availability passively, reading the environment only, which is correct for startup and catalog callers and wrong for a status the user asked for: a key kept in car secrets rather than exported is invisible to it, so the table reports a usable model as RUNNABLE no. Measured with OPENAI_API_KEY in the keychain and nothing in the environment, this path said openai/gpt-5.5 was not runnable while inference through the same binary worked.

Refreshing the registry before calling the ordinary path does NOT fix that, and the reason is worth stating because it is invisible at the call site: this list clones the registry and re-refreshes the clone, so any authoritative state a caller established is discarded on the way in. The mode has to be chosen here.

Costs one secret-store read per DISTINCT credential name, deduplicated inside the refresh — not one per model.

pub fn retire_plan(&self, model_id: &str) -> Result<RetirePlan, InferenceError>

What retiring model_id would delete — CAR’s projections under the models directory and the hub content only CAR fetched — and anything that stops it. Reads only; see retire::plan_hub_retirement.

pub fn retire_plan_in( &self, model_id: &str, hub: &Path, ) -> Result<RetirePlan, InferenceError>

Self::retire_plan against an explicit hub directory.

pub async fn retire_model( &self, model_id: &str, dry_run: bool, expect: Option<&str>, ) -> Result<RetireOutcome, InferenceError>

Retire model_id: delete what Self::retire_plan says is CAR’s — the hub content only CAR fetched and its own projections — leaving the model re-fetchable on next use. dry_run returns the plan and deletes nothing; expect (a dry run’s retire::RetirePlan::digest) refuses execution unless the plan made under the locks is the one that was reviewed.

pub async fn retire_model_in( &self, model_id: &str, hub: &Path, dry_run: bool, expect: Option<&str>, ) -> Result<RetireOutcome, InferenceError>

Self::retire_model against an explicit hub directory.

Never executes a stored plan and never waits on someone else’s work: it takes every download lock the first plan names without blocking (refusing downloading if one is busy), plans again under them, holds every retired row’s maintenance guard — no new load, worker unloaded — across the whole destructive window, and only then deletes: projections first (while the weights they point at still exist), then hub content. Each path is re-checked immediately before it goes and unlinked relative to a descriptor for its directory; a whole repo is journaled, detached into a quarantine beside it and removed by descriptor; a receipt-backed projection goes through the journaled model removal, its tombstone cleared after under a journaled intent, so the row ends not installed but fetchable. Execution stops at the first error, and what it already removed — or detached — is reported.

pub fn set_external_pins(&self, pins: BTreeMap<String, Vec<String>>)

Replace the models declarative agents pin (id or name → the names of the agents pinning it). The daemon refreshes this from the declarative-agent registry before assessing or retiring, so an agent that runs monthly keeps its model. Pushed, not pulled: anything that assesses or retires without going through those daemon handlers — an autonomous executor — must refresh the pins first, or it plans against stale ones.

pub fn start_usage_tracking(&self)

Begin tracking local-model usage on this machine, if it has not begun. Daemon startup only; see model_management::ModelManagementStore::start_usage_tracking.

pub async fn set_resource_policy_by_user( &self, policy: ResourcePolicy, ) -> Result<(), ResourcePolicyError>

Save and apply policy as the user’s explicit choice — the concierge’s marker cleared first, so a saved choice can never read as the concierge’s (a crash after this point leaves an unmarked policy, which reads as chosen) — under the lock the concierge’s sizing holds.

pub fn usage_heartbeat(&self)

Record that this daemon, which stamps local-model use, is up. The daemon calls it every 15 minutes; see model_management::ModelManagementStore::beat.

pub fn effective_update_policy( &self, prefs: &UpdatePreferences, start_clock: bool, ) -> (UpdatePolicy, Option<u64>)

The update policy in force: prefs.policy, except that the unchosen default Auto behaves as Notify for its first week — suggesting upgrades and retirements without applying any — so a user who wanted notify can still say so. Returns when that week ends while it runs. start_clock records the week’s start on first sight (the daemon’s maintenance pass); other readers only consult it.

pub async fn maintenance_pass(&self, dry_run: bool) -> MaintenanceReport

One pass of autonomous local-model maintenance. Under the update policy (Off does nothing, Notify only suggests, Auto acts): finish interrupted retirements, assess the portfolio, decide with maintenance::decide — useless models whenever, replaced ones after a month or under disk pressure, idle ones only under disk pressure, never one the user kept or fetched back after a retirement — retire at most one with its reviewed plan digest, and discard abandoned partial downloads from repos CAR owns. dry_run decides and deletes nothing.

pub async fn maintenance_pass_in( &self, hub: &Path, hw: &HardwareInfo, dry_run: bool, ) -> MaintenanceReport

Self::maintenance_pass against an explicit hub and hardware.

pub async fn maintenance_pass_with( &self, hub: &Path, hw: &HardwareInfo, dry_run: bool, policy: UpdatePolicy, nudge: &NudgeState, disk: Option<DiskSpace>, ) -> MaintenanceReport

Self::maintenance_pass_in with the update policy, the user’s dismissals and the disk reading given rather than read.

pub async fn portfolio(&self) -> Portfolio

Assess every installed local model: what it is doing for this machine and whether it is worth its disk. Reads only. See portfolio.

pub async fn portfolio_in(&self, hub: &Path, hw: &HardwareInfo) -> Portfolio

Self::portfolio against an explicit hub and hardware.

pub async fn resume_retirements(&self) -> Vec<String>

Finish retirements a crash interrupted — journaled hub quarantines and tombstones a receipt-backed retirement set but did not clear. Daemon start only, before the listener binds: run beside a live retirement it would race that retirement’s own journal and walk.

pub fn model_management_store(&self) -> &ModelManagementStore

pub fn available_model_upgrades(&self) -> Vec<ModelUpgrade>

Report installed models that have curated newer replacements.

pub async fn check_upgrade_nudge( &self, inference_active: bool, ) -> (NudgeDecision, NudgeState)

The proactive-upgrade decision for right now: which curated upgrades to auto-apply (under Auto policy) and the single nudge to surface, with throttling and dismissals applied. The daemon calls this on its periodic check and broadcasts decision.nudge over WebSocket. Returns the loaded NudgeState too so the caller can stamp last_nudge_secs after sending.

pub fn dismiss_upgrade_nudge( &self, dismiss_key: &str, ) -> Result<(), InferenceError>

Record that the user dismissed a nudge (by its dismiss_key), so it is never surfaced again. Persists to ~/.car/nudge-state.json.

pub async fn check_concierge( &self, inference_active: bool, ) -> (Vec<ConciergeSuggestion>, NudgeState)

Run the proactive concierge decision: for the default watched lanes, suggest a model to acquire for any lane the user has nothing installed for. Returns the suggestions plus the loaded NudgeState so the caller can stamp last_concierge_secs after surfacing (mirrors the stamp-after-deliver pattern of Self::check_upgrade_nudge). The concierge throttles independently of the upgrade nudge.

pub fn dismiss_concierge_suggestion( &self, dismiss_key: &str, ) -> Result<(), InferenceError>

Record that the user dismissed a concierge suggestion (by its dismiss_key), so it is never surfaced again. Shares the same ~/.car/nudge-state.json dismissed list as the upgrade nudge — the key namespaces are disjoint (concierge:… vs from=>to).

pub fn dismiss_concierge_labeled( &self, dismiss_key: &str, reason: DismissReason, ) -> Result<(), String>

Record a labeled concierge dismissal (Phase B4/C1) so the Act gate can treat the reason as signal (permanent reasons suppress; NotNow cools down). Persists to ~/.car/nudge-state.json.

pub async fn check_canaries(&self) -> Vec<UseCase>

Net-positive verification (Phase F3): for each lane whose latest action was a SetDefault (a switch not yet rolled back), compare the new model’s observed post-switch success in that lane against the prior model’s baseline; if it’s measurably worse with enough samples, auto-revert to the prior. Never self-graded — the signal is the outcome ledger’s verifier/outcome receipts. The daemon calls this on its periodic tick. Returns the reverted lanes.

pub async fn concierge_ask( &self, question: Option<&str>, ) -> Result<ConciergeAnswer, String>

Conversational concierge (Phase F1/F2): answer a free-form question about the user’s models/portfolio, grounded in the observed-usage evidence + the deterministic recommend() candidate menu. The LLM explains — it runs on a local model, is told to answer ONLY from the supplied evidence, and must not invent a model or assert fit (the grounding oracle already decided fit). This is the ModelConcierge “agent”: a thin, constrained generate call over assembled receipts, not a freelancing chat. Answer a question about this machine’s local models. The facts come first and need no model: concierge_answer::summarize states the portfolio, what maintenance would do, the memory policy, and what the concierge did on its own. With a question, a model answers from that evidence plus observed usage — routed for quality over latency, on device when one can, bounded in length, cut at its first repeated line, and with every model id it names checked against the evidence. If the model fails, the facts are still the answer.

pub async fn refresh_catalog(&self) -> Result<usize, String>

Refresh the model catalog from the configured signed source (Phase E1): fetch + verify (detached ed25519 against the pinned key), cache it if it is newer, and load it into the running registry — the new models surface as recommend() candidates and concierge suggestions without a restart. Source is catalog::catalog_url and catalog::catalog_public_key — car.parslee.ai and its key unless CAR_CATALOG_URL / CAR_CATALOG_PUBKEY say otherwise. Returns the number of models in the verified catalog; an older or equal version is an error.

pub fn mark_catalog_loop_alive(&self)

Record the catalog loop’s tick. The daemon’s loop only.

pub fn mark_daemon_alive_for_catalog(&self)

Record this home’s daemon heartbeat beside the loop’s tick. The daemon’s heartbeat task only, which is not the catalog loop.

pub fn catalog_freshness(&self) -> CatalogFreshness

Whether signed-catalog updates are reaching this machine, from the recorded checks. Reads only; never fetches.

pub async fn refresh_catalog_on_schedule(&self) -> Result<Option<usize>, String>

The daemon’s periodic refresh: like Self::refresh_catalog, but a catalog no newer than the cached one is Ok(None) rather than an error — the usual answer between publishes — and the off update policy fetches nothing. Ok(Some(n)) when a newer catalog loaded.

pub async fn enforce_catalog_revocations(&self)

Apply the cached catalog’s revocations to lane defaults now: run on every maintenance pass, so a daemon that restarted with a revocation already cached — or a default set by a path that predates this check — still has the lane cleared.

pub async fn discover_models(&self) -> Result<usize, String>

Auto-discover provider models (Phase E2): query the provider’s /v1/models list and cache previously-unknown chat/reasoning models as TrustTier::Community entries (cloning a curated same-provider schema as a template). Best-effort — no key or no OpenAI provider configured is a no-op, not an error. Discovered models load into the registry at next startup; unlike the signed catalog, discovery is not loaded live. Returns the total number of cached discovered models. This is what lets the catalog (and the router) pick up new models like a gpt-5.5 without a release.

pub fn lane_defaults(&self) -> LaneDefaults

All configured lane defaults (Phase D1), from the in-memory cache.

pub fn lane_default( &self, project: Option<&str>, use_case: UseCase, ) -> Option<String>

Resolve the default model for (project, use_case), if set. Routing consults this as a strong preference before falling back to adaptive selection. Reads the cache (no disk).

pub fn set_lane_default( &self, project: Option<String>, use_case: UseCase, model_id: &str, ) -> Result<(), String>

Set the default model for (project, use_case) (Phase D1) — the durable target of the concierge’s “set it up” action.

pub fn clear_lane_default( &self, project: Option<&str>, use_case: UseCase, ) -> Result<bool, String>

Clear the default for (project, use_case). Returns whether one existed (used by rollback in D3).

pub async fn user_set_lane_default( &self, use_case: UseCase, model_id: &str, project: Option<String>, ) -> Result<(), String>

User-facing lane-default set (the concierge.set_default WS path): like set_lane_default but serialized under the concierge action lock AND recorded in the action ledger. Without the ledger entry a manual pin would be invisible to the canary, which could then revert it based on a stale ledger view — so this records a SetDefault (with the prior captured) exactly like apply, keeping the ledger and the live default consistent.

pub async fn user_clear_lane_default( &self, use_case: UseCase, project: Option<String>, ) -> Result<bool, String>

User-facing lane-default clear (the concierge.clear_default WS path): serialized + ledgered like user_set_lane_default. Records a ClearDefault so the canary sees the lane is no longer a standing switch (its latest action is the clear, not a SetDefault).

pub fn concierge_actions(&self, limit: usize) -> Vec<ConciergeActionEntry>

The recorded concierge actions (Phase D2), most recent last.

pub async fn apply_concierge( &self, use_case: UseCase, model_id: &str, project: Option<String>, ) -> Result<ConciergeApplyResult, String>

Closed-loop “set it up” (Phase D3): acquire model_id, then set it as the lane default — capturing the prior default so the change is reversible (rollback_lane). Every step is recorded in the action ledger.

Consent: the caller (CarHost) owns the pre-download confirmation — this primitive assumes the user has already agreed to the (possibly multi-GB) download; the ledger entry is the audit record that it happened. Single-writer: lane defaults assume one concierge writer (CarHost); concurrent applys would last-write-wins the JSON (the F3 canary watcher must coordinate before it becomes a 2nd writer).

pub async fn rollback_lane( &self, use_case: UseCase, project: Option<String>, ) -> Result<Option<String>, String>

Revert a lane default to its value before the last apply (Phase D3 rollback). Restores the prior model (or clears the default if there was none), recording the rollback. Returns the restored model id, or None if the default was cleared / nothing to undo.

pub async fn concierge_status(&self, inference_active: bool) -> ConciergeStatus

Assemble the ambient concierge status (Phase C1): per-lane usage + friction from the outcome ledger, the current grounded decision (evaluate_concierge), and per-model health from the profiles. A pull (the UI asks); proactive push stays separate.

pub async fn detect_upgrades(&self) -> Vec<UpgradeFinding>

Detect upgrades combining curated rules with upstream Hub discovery, honoring update preferences (channel/policy) and the TTL cache. Upstream probing only happens on the Latest channel and is offline-safe.

pub fn list_schemas(&self) -> Vec<ModelSchema>

List all known models and their download status (legacy). List all model schemas from the unified registry (full metadata).

pub fn catalog_snapshot(&self) -> Result<CatalogSnapshot, String>

Deterministic immutable catalog view used to bind inference routing to exact model rows. Runtime availability never participates in either row digests or the catalog revision.

pub fn registered_schema(&self, id: &str) -> Option<ModelSchema>

Return one registered schema without refreshing availability.

This is for identity/provenance checks that must reflect signed catalog overrides while remaining local and side-effect free.

pub fn list_models(&self) -> Vec<ModelInfo>

pub fn knows_model(&self, name: &str) -> bool

Whether a caller-supplied model name resolves to a registered schema.

Answers the question generation asks, by the same two routes and in the same order: exact id, then the case-insensitive name lookup. It is deliberately NOT list_models(), which returns the on-device catalog — checking a remote model id against that set reports every cloud model as unknown.

Exists so a caller that fans out to several named models can refuse a typo up front instead of discovering it as a generation error per request. Says nothing about whether the model is currently reachable (credentials, network) — only that the name is one CAR knows.

pub fn model_schema(&self, name: &str) -> Option<ModelSchema>

The registered schema behind a model name or id, resolved exactly as Self::knows_model resolves it — exact id first, then the case-insensitive name lookup.

Defined together with knows_model so the two cannot drift into disagreeing about which names exist, and it resolves in the SAME order generation does (get(id).or_else(find_by_name(id)), as at the routing sites) — so a caller asking “will these two names reach the same model?” gets the answer that will actually hold at generation time, including find_by_name’s MLX-variant redirect on Apple silicon.

That fidelity is the point, and it is NOT a canonical identity oracle. The lookup is over a HashMap, so if two rows share a display name the one returned is arbitrary — stable within a process, not across restarts. Generation has the same property, so a caller comparing what will run stays correct; a caller needing a stable identity for storage wants the exact id via registered_schema.

pub async fn pull_model(&self, name: &str) -> Result<PathBuf, InferenceError>

Download a model if not already present.

pub async fn pull_model_with_progress( &self, name: &str, sink: &ProgressSink, ) -> Result<PathBuf, InferenceError>

Download a model if not already present, reporting progress to sink and enforcing the acquisition lifecycle (per-model lock, disk preflight, lifecycle events). The CLI and daemon use this to show live progress. Resolves and installs against one registry generation.

pub async fn adopt_model_into_car( &self, model_id: &str, ) -> Result<InstallReceipt, InferenceError>

Explicitly adopt an already-usable local artifact into CAR ownership. The source path is resolved from the registry; callers cannot nominate an arbitrary deletion target.

pub async fn remove_model_from_car( &self, model_id: &str, ) -> Result<RemoveFromCarResult, InferenceError>

Safely remove only CAR-owned linkage after every Task 3 runtime owner acknowledges release. Shared Hugging Face blobs remain untouched.

pub fn update_prefs(&self) -> UpdatePreferences

Current update preferences. A team-shared project .car/update-prefs.json (found by walking up from cwd) overrides the user ~/.car/update-prefs.json; defaults if neither exists. Loaded on demand — read at onboarding/ upgrade-check frequency, not on the inference hot path.

pub fn set_update_prefs( &self, prefs: &UpdatePreferences, ) -> Result<(), InferenceError>

Persist update preferences to ~/.car/update-prefs.json.

pub fn remove_model(&self, name: &str) -> Result<(), InferenceError>

👎Deprecated:

use async remove_model_from_car for receipt-backed safe removal

Legacy synchronous removal is intentionally disabled because it cannot coordinate active workers, cross-process leases, or receipt ownership. Use Self::remove_model_from_car instead.

pub fn register_model(&mut self, schema: ModelSchema)

Register a model at the public runtime boundary.

The registry normalizes every such schema to Community trust. Project curation is reserved for compiled builtins and signature-verified catalogs inside this crate.

pub fn register_user_model(&mut self, schema: ModelSchema)

Register a model from a user-controlled schema boundary.

pub async fn discover_vllm_mlx_models(&mut self) -> usize

Discover generic MLX models from a running vLLM-MLX server and register them. Returns the number of discovered models added or refreshed in the registry.

pub fn outcome_tracker(&self) -> Arc<RwLock<OutcomeTracker>>

Get outcome tracker for external use (e.g., memgine integration).

pub async fn save_outcomes(&self) -> Result<(), Error>

Persist outcome profiles to disk for cross-session learning (#13). Unconditional (force) save — writes even if nothing changed. Prefer flush_outcomes for shutdown / periodic flushes; the per-call path uses auto_save_outcomes, which debounces.

pub async fn flush_outcomes(&self) -> Result<bool, Error>

Flush outcome profiles to disk iff dirty, ignoring the per-call time debounce. Returns whether a write happened. This is the durable-receipt backstop: the daemon calls it on a periodic timer and on graceful shutdown so the last (sub-OUTCOME_FLUSH_INTERVAL) window of learning is never lost. Cheap when clean (no write).

pub async fn reconcile_outcome_ledger( &self, ) -> Result<LedgerReconciliation, Error>

Make the outcome ledger and the persisted profiles agree about which failures were the model’s — see outcome::plan_reconcile.

The daemon calls this once at startup, before its ticks run. It is deliberately not part of engine construction: every test, benchmark and CLI process builds an engine, many on the real state root, and a rewrite of the ledger belongs to the one process that owns it (the same one that prunes it). Skipped in offload workers, when the ledger is disabled, and when the profiles file existed but could not be read.

Order is the crash guarantee: pending receipts are flushed first, the stamped ledger is written, and only then are the corrected profiles committed and saved. A failed write leaves the tracker untouched; a crash after it leaves those failures counted — the conservative direction. Neither can take a failure back twice.

pub async fn prune_outcome_ledger(&self, max_entries: usize) -> Result<()>

Enforce the outcome-ledger retention bound (privacy + disk). A cheap no-op when under the cap; the daemon calls it periodically.

pub async fn save_key_pool_stats(&self) -> Result<(), Error>

Persist key pool stats to disk.

pub async fn key_pool_stats(&self) -> HashMap<String, Vec<KeyStats>>

Get key pool stats for all endpoints.

pub async fn export_profiles(&self) -> Vec<ModelProfile>

Export model performance profiles for persistence.

pub fn outcome_scoreboard(&self) -> Scoreboard

Fold the durable outcome ledger into the deployment scoreboard — the per-model, priced, OUTCOME-DENOMINATED view (cost-per-success, tokens-per-success, success-rate). Reads the same outcome_ledger.jsonl the tracker flushes to (cross-session, survives restart) and joins per-model catalog prices from the registry so usd_per_success is the honest “cry once” figure. Unpriced models keep a None dollar figure rather than a fabricated one. See crate::scoreboard::Scoreboard.

pub async fn import_profiles(&self, profiles: Vec<ModelProfile>)

Import model performance profiles (from persistence).

pub async fn prepare_speech_runtime(&self) -> Result<PathBuf, InferenceError>

Ensure the managed local speech runtime exists and return its root directory — the same root speech_health reports, on every platform.

Apple Silicon used to short-circuit here: native MLX backends were taken to replace the Python runtime outright, so this only created models_dir and handed that back without ever provisioning the managed runtime. Since #640 the runtime is a live fallback there too (the native backends can’t load every catalogued checkpoint) and speech doctor reports its real state — so a “successful” install contradicted doctor, printed a path doctor never mentions, and pushed the multi-minute venv+pip bootstrap onto the first synthesis (Parslee-ai/car#649). Provision it up front on every platform instead.

The one asymmetry that remains is what a bootstrap failure means. Off Apple Silicon the managed runtime is the only local speech path, so failing to build it fails the call. On Apple Silicon it sits behind working native backends, so a missing uv degrades rather than breaks: the root comes back either way, and callers should report speech_health().runtime.installed rather than read a returned path as proof of success. Either way the returned directory exists — a method called “prepare” leaves the thing prepared (Parslee-ai/car#626).

pub fn set_speech_policy(&mut self, policy: SpeechPolicy)

Override speech routing preferences for the current engine instance.

pub fn set_routing_config(&mut self, config: RoutingConfig)

pub async fn install_curated_speech( &mut self, ) -> Result<Vec<SpeechInstallReport>, InferenceError>

Download the curated local speech model set into the shared Hugging Face cache.

pub fn speech_health(&self) -> SpeechHealthReport

Report speech runtime, model cache, and remote-provider health.

pub async fn model_health(&self) -> ModelHealthReport

Report the current model catalog, configured defaults, capability coverage, and speech runtime/provider health in one place.

pub async fn model_health_for_explicit_status(&self) -> ModelHealthReport

Self::model_health for a surface a person explicitly asked for a status from (car doctor), resolving credential-backed availability authoritatively.

Without it the report’s available_models counts come from the passive environment-only refresh, so a provider whose key lives in car secrets reports zero available models beside configured=yes — the diagnostic a person runs to find out why something is not working, telling them the working thing does not work.

pub async fn smoke_test_speech( &self, local: bool, remote: bool, ) -> Result<SpeechSmokeReport, InferenceError>

Run a real speech smoke test through the configured local and/or remote paths.

Trait Implementations

impl InferenceHandle for InferenceEngine

fn generate<'life0, 'async_trait>( &'life0 self, req: GenerateRequest, ) -> Pin<Box<dyn Future<Output = Result<String, InferenceError>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait,

Run a generation request to completion. Same contract as InferenceEngine::generate: caller passes a GenerateRequest (which may carry an explicit model, a routing hint, tools, or a thinking budget), receives the final text or an InferenceError.

fn embed<'life0, 'async_trait>( &'life0 self, req: EmbedRequest, ) -> Pin<Box<dyn Future<Output = Result<Vec<Vec<f32>>, InferenceError>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait,

Encode one or more texts as embedding vectors. Same contract as InferenceEngine::embed: returns one Vec<f32> per input text in the same order.