Struct InferenceEngine 

pub struct InferenceEngine {
    pub config: InferenceConfig,
    pub unified_registry: UnifiedRegistry,
    pub adaptive_router: AdaptiveRouter,
    pub outcome_tracker: Arc<RwLock<OutcomeTracker>>,
    pub registry: ModelRegistry,
    pub router: ModelRouter,
    /* private fields */
}

The main inference engine. Thread-safe, lazily loads models.

Now includes the unified registry, adaptive router, and outcome tracker for schema-driven model selection with learned performance profiles.

Fields

config: InferenceConfig

unified_registry: UnifiedRegistry

Unified model registry (local + remote).

adaptive_router: AdaptiveRouter

Adaptive router with three-phase selection.

outcome_tracker: Arc<RwLock<OutcomeTracker>>

Outcome tracker for learning from results.

registry: ModelRegistry

router: ModelRouter

Implementations

impl InferenceEngine

pub fn local_model_retention(&self, model_id: &str) -> BackendRetention

Actual post-dispatch retention observed by worker/process owners. Success alone is insufficient: zero-cache loads remain transient.

pub fn local_model_preflight( &self, model_id: &str, context_tokens: usize, ) -> Result<LocalLoadPreflight, InferenceError>

Evaluate one local model without downloading or loading it.

pub fn active_local_resource_policy(&self) -> ResourcePolicyLoadEvidence

The policy currently enforced by local-model admission, plus the load source/warning captured when this engine initialized it.

Read-side model surfaces use this accessor rather than reopening the policy file, so a policy applied to the running engine is one atomic source of truth for preflight, fit annotations, and recommendations.

pub fn apply_local_resource_policy(&self, policy: ResourcePolicy)

Update the in-memory admission/cache ceiling after persistence succeeds. Idle entries are reclaimed by each cache’s next sweep/access; active inference is never killed by a policy decrease.

pub fn begin_local_model_maintenance( &self, model_id: &str, ) -> Result<LocalModelMaintenanceGuard, ModelMaintenanceError>

pub async fn prepare_local_model_removal( &self, model_id: &str, ) -> Result<LocalModelMaintenanceGuard, ModelMaintenanceError>

pub fn evict_local_model_if_idle(&self, model_id: &str) -> bool

Targeted cache eviction for safe model removal. Callers must hold the per-model maintenance guard while invoking this and checking any worker/cross-process leases.

pub fn set_spend_limits(&self, limits: Option<SpendLimits>)

Install (or clear) spend limits (I4). per_request_usd also arms the mid-stream guard on streaming calls: the stream is cancelled with a terminal StopReason("spend_limit: ...") the moment the estimated running cost (prompt + streamed output) crosses the budget.

pub fn new(config: InferenceConfig) -> Self

pub async fn init_key_pool(&self)

Initialize key pool: register keys from all remote models and load persisted stats. Call this after construction (requires async).

pub async fn reset_local_kv_cache(&self, model_id: &str)

Clear the in-process KV / prefix cache of a loaded local model.

Prefix reuse (begin_prompt) is a per-conversation optimization: it reuses the KV state of a shared token prefix across calls. When one engine is driven through a sequence of independent prompts (e.g. a benchmark’s task suite), that reuse leaks decode state between unrelated conversations — and reusing cached KV instead of a fresh prefill introduces tiny numerical drift that can flip a greedy (temperature-0) token, making multi-step runs non-reproducible. Calling this between independent runs restores a clean slate. No-op for remote models or a backend that isn’t currently loaded.

pub async fn warm_up<S: AsRef<str>>( &self, _schema_ids: &[S], ) -> Vec<Result<(), InferenceError>>

No-op on non-macOS — MLX doesn’t run here.

pub async fn evict_idle_vllm_servers(&self) -> usize

Stop idle supervised vllm-mlx servers. Driven by the same idle loop that evicts in-process backends; returns the number stopped.

pub fn evict_idle_backends(&self) -> (usize, u64)

No-op on non-macOS — there are no accumulating backend caches.

pub async fn route_adaptive(&self, prompt: &str) -> AdaptiveRoutingDecision

Route a prompt using the adaptive router (new). Returns full decision context.

pub async fn route_adaptive_with_intent( &self, prompt: &str, intent: Option<IntentHint>, ) -> AdaptiveRoutingDecision

Like route_adaptive but honors a caller IntentHint — notably exclude_models for adversarial-reviewer separation: “route me any capable model that is NOT the one that just did the work” (car#358). An excluded id is never chosen while any non-excluded capable model exists — including via the preferred-model override (skipped when it names an excluded model) and the cold-start fallbacks. The exclusion is soft: if excluding leaves nothing routable, an excluded model may still be returned as a last resort (a same-model review beats no review).

pub fn route(&self, prompt: &str) -> RoutingDecision

Route a prompt to the best model without executing (legacy compat).

pub fn estimated_tokens( &self, req: &GenerateRequest, model_id: Option<&str>, ) -> (usize, usize, bool)

Estimate token count for a request against a specific model’s context window. Returns (estimated_input_tokens, context_window_tokens, fits).

Multimodal content blocks (image/video/audio, in images or in messages history) contribute provider-calibrated estimates via media_tokens — a minute of video is ~15.8K input tokens at Gemini’s documented rate, not zero — and the multi-turn messages history’s text is counted too (chars/4), not just its media. This feeds the adaptive router’s window-fit / needs_compaction signal.

pub fn model_context_window(&self, model_id: &str) -> usize

The model’s context window in tokens, or 0 if the id is unknown (unregistered). Public so a multi-turn driver (e.g. the assistant loop) can bound its running message history to the window before it overflows — an overflowed history pushes the model to its context limit and can truncate the original task provider-side.

pub async fn generate_tracked( &self, req: GenerateRequest, ) -> Result<InferenceResult, InferenceError>

Generate text with full tracking (tool_calls, usage, trace_id, latency, TTFT), plus Qwen3 hybrid-thinking recovery.

Qwen3 (and other hybrid-thinking models) default to reasoning ON. With a small max_tokens budget the model can spend the entire budget inside an unclosed <think> block, so the strip pass returns empty text — infer(prompt, model, 16) then silently yields “” while a non-thinking model answers fine (car-releases#60, #62).

When the caller left thinking on Auto (didn’t explicitly opt into reasoning) and nothing usable came back, retry once with reasoning suppressed so the caller gets a direct answer — matching the CLI’s --thinking off default, but for every FFI/daemon path. Either way, record why via stop_reason so an empty result is never silent.

pub async fn generate_tracked_stream( &self, req: GenerateRequest, ) -> Result<TrackedStream, InferenceError>

Stream a generation and record an outcome when it finishes.

Wraps generate_stream_raw with a forwarding “tap” task: it accumulates the event stream (via stream::StreamAccumulator), forwards every event to the caller unchanged, and on completion books a success/failure against the model’s profile — the same outcome telemetry the non-streaming generate_tracked path records. Without this, every streamed inference (voice/realtime, the daemon’s infer stream) was invisible to model-health scoring. Cancellation propagates: if the caller drops the returned receiver, the tap stops forwarding, drops the producer receiver, and the backend task observes the closed channel.

Returns a TrackedStream carrying the resolved model + this call’s trace_id alongside the event receiver. The trace_id is known up front (minted by record_start before the first token), so a caller can score the finished turn against the same trace the tap will resolve — the streaming counterpart of the non-streaming path’s InferenceResult {trace_id, model_used}, which the conversation-outcome signal needs.

The events receiver yields StreamEvent variants (TextDelta, ToolCallStart, ToolCallDelta, Usage, StopReason, ProviderOutputItem, Error, Done); use a stream::StreamAccumulator to collect them into a final result. Local backends (MLX, Candle) emit true incremental TextDeltas per token, enabling token-by-token UI, overlapping TTS, and early cancellation. The channel buffers 64 events so burst tokens don’t block generation.

Example: voice app integration

let mut handle = engine.generate_tracked_stream(req).await?;
let mut text_buf = String::new();
while let Some(event) = handle.events.recv().await {
    match event {
        StreamEvent::TextDelta(delta) => {
            text_buf.push_str(&delta);
            // Feed text_buf to TTS when a sentence boundary is reached
        }
        StreamEvent::Done { text, .. } => break,
        _ => {}
    }
}
// handle.trace_id / handle.model_used identify the turn for scoring.

pub async fn route_context_snapshot( &self, prompt: &str, workload: RoutingWorkload, has_tools: bool, has_vision: bool, ) -> AdaptiveRoutingDecision

Route a prompt using the adaptive router without executing inference.

pub async fn generate( &self, req: GenerateRequest, ) -> Result<String, InferenceError>

Generate text from a prompt (legacy API, no outcome tracking). When req.model is None, uses intelligent routing based on prompt complexity.

pub async fn tokenize( &self, model: &str, text: &str, ) -> Result<Vec<u32>, InferenceError>

Encode text via the named model’s tokenizer. Returns raw token IDs without any chat-template wrapping or BOS-prepending — pair with Self::detokenize for the round-trip property detokenize(model, tokenize(model, s)) == s for any UTF-8 s.

Only local models have a tokenizer the runtime can call directly (Candle/GGUF on Linux/Windows, MLX on Apple Silicon). For remote models the call returns InferenceError::UnsupportedMode — provider tokenizer endpoints vary too widely to be portable here, and bundling tiktoken-style tables would lock the registry to a fixed set of providers.

pub async fn detokenize( &self, model: &str, tokens: &[u32], ) -> Result<String, InferenceError>

Inverse of Self::tokenize: decode token IDs back to text.

pub async fn embed( &self, req: EmbedRequest, ) -> Result<Vec<Vec<f32>>, InferenceError>

Generate embeddings for text using the dedicated embedding model. On Apple Silicon, uses the native MLX backend; on other platforms, uses Candle.

pub async fn rerank( &self, req: RerankRequest, ) -> Result<RerankResult, InferenceError>

Rerank candidate documents against a query using a cross-encoder reranker model (Qwen3-Reranker family). Returns documents sorted by descending relevance.

Scoring

Qwen3-Reranker is a Qwen3 base LM fine-tuned so that the first assistant token is "yes" or "no" given the templated <Instruct>/<Query>/<Document> user turn. We run a short greedy decode (≤ 3 tokens, so a leading space, BOS artifact, or the occasional newline don’t break us) and score yes → 1.0, no → 0.0, anything else → 0.5 with a warning.

This is a binary score — the soft probability softmax(logit_yes, logit_no) would give finer ordering but requires per-token logit access on backend::MlxBackend, which isn’t exposed publicly yet. Tracked as a follow-up; binary scores still produce a correct partial ordering, just with coarser tiebreaks within the {yes} or {no} groups.

Prompt template

We emit the upstream Qwen3-Reranker chat template verbatim: a dedicated system prompt fixing the yes/no answer space, then the user turn with <Instruct>/<Query>/<Document>, then the assistant prefix with a closed empty <think> block to suppress thinking (reranker is not a reasoner — it’s a classifier). Deviating from this template produces sharply degraded yes/no distributions.

pub async fn ground( &self, req: GroundRequest, ) -> Result<GroundResult, InferenceError>

Dedicated endpoint for structured visual grounding.

Runs a VL generate call under the hood and parses Qwen2.5-VL’s inline <|object_ref_*|>...<|box_*|>(x1,y1),(x2,y2) spans into typed BoundingBoxes. Distinct from the generic InferenceEngine::generate + InferenceResult.bounding_boxes path so callers can express “I want boxes” as a first-class intent — which also lets the router prefer models that declare the Grounding capability.

pub async fn classify( &self, req: ClassifyRequest, ) -> Result<Vec<ClassifyResult>, InferenceError>

Classify text against candidate labels. When req.model is None, routes to the smallest available model.

pub async fn transcribe( &self, req: TranscribeRequest, ) -> Result<TranscribeResult, InferenceError>

Transcribe an audio file using the best available STT model.

pub async fn synthesize( &self, req: SynthesizeRequest, ) -> Result<SynthesizeResult, InferenceError>

Synthesize speech using the best available TTS model.

pub async fn generate_image( &self, req: GenerateImageRequest, ) -> Result<GenerateImageResult, InferenceError>

Generate an image using the best available local MLX image model.

pub async fn generate_image_batch( &self, req: GenerateImageRequest, ) -> Result<Vec<GenerateImageResult>, InferenceError>

Generate one or more variants in a single call.

Returns req.variant_count results (defaulting to 1). The current MLX Flux backend doesn’t support native batching, so this loops over generate_image with the seed advanced per variant for visual diversity. A future hosted backend (gpt-image-2, Replicate) can short-circuit this with one network call producing N coherent images.

Per-variant errors abort the batch — there’s no partial- success semantics today. Callers needing more lenient behaviour should call generate_image directly in their own loop.

Closes #110.

pub async fn generate_video( &self, req: GenerateVideoRequest, ) -> Result<GenerateVideoResult, InferenceError>

Generate a video using the best available local MLX video model.

pub fn response_format_rejection_reason( &self, model: &str, rf: &ResponseFormat, ) -> Option<String>

List all known models and their status (new registry). Why model cannot honor a response_format, or None when it can (or is unknown here). Asks the SAME protocol handler the remote path consults (ProtocolHandler::supports_response_format) so a CLI can warn before a run rather than discover the UnsupportedMode on the repair turn. The Parslee gateway rejects every format separately in execute_request, so it is named here too.

pub fn list_models_unified(&self) -> Vec<ModelInfo>

The unified catalog, annotated for the machine this engine runs on under its active local-model resource policy — the same policy models.preflight admits against, kept in step with the persisted one by apply_local_resource_policy.

pub fn model_fit(&self, schema: &ModelSchema) -> ModelFit

The fit annotation for one schema on this machine under the active policy — list_models_unified’s verdict for a row built elsewhere (models.search), so every catalog surface publishes the same one.

pub fn list_models_unified_for( &self, hardware: &HardwareInfo, policy: &ResourcePolicy, ) -> Vec<ModelInfo>

Self::list_models_unified against explicit hardware and policy. Every row is returned in registry order with every existing field unchanged; the fit annotation is computed per call and never stored.

pub fn model_management_store(&self) -> &ModelManagementStore

pub fn available_model_upgrades(&self) -> Vec<ModelUpgrade>

Report installed models that have curated newer replacements.

pub async fn check_upgrade_nudge( &self, inference_active: bool, ) -> (NudgeDecision, NudgeState)

The proactive-upgrade decision for right now: which curated upgrades to auto-apply (under Auto policy) and the single nudge to surface, with throttling and dismissals applied. The daemon calls this on its periodic check and broadcasts decision.nudge over WebSocket. Returns the loaded NudgeState too so the caller can stamp last_nudge_secs after sending.

pub fn dismiss_upgrade_nudge( &self, dismiss_key: &str, ) -> Result<(), InferenceError>

Record that the user dismissed a nudge (by its dismiss_key), so it is never surfaced again. Persists to ~/.car/nudge-state.json.

pub async fn check_concierge( &self, inference_active: bool, ) -> (Vec<ConciergeSuggestion>, NudgeState)

Run the proactive concierge decision: for the default watched lanes, suggest a model to acquire for any lane the user has nothing installed for. Returns the suggestions plus the loaded NudgeState so the caller can stamp last_concierge_secs after surfacing (mirrors the stamp-after-deliver pattern of Self::check_upgrade_nudge). The concierge throttles independently of the upgrade nudge.

pub fn dismiss_concierge_suggestion( &self, dismiss_key: &str, ) -> Result<(), InferenceError>

Record that the user dismissed a concierge suggestion (by its dismiss_key), so it is never surfaced again. Shares the same ~/.car/nudge-state.json dismissed list as the upgrade nudge — the key namespaces are disjoint (concierge:… vs from=>to).

pub fn dismiss_concierge_labeled( &self, dismiss_key: &str, reason: DismissReason, ) -> Result<(), String>

Record a labeled concierge dismissal (Phase B4/C1) so the Act gate can treat the reason as signal (permanent reasons suppress; NotNow cools down). Persists to ~/.car/nudge-state.json.

pub async fn check_canaries(&self) -> Vec<UseCase>

Net-positive verification (Phase F3): for each lane whose latest action was a SetDefault (a switch not yet rolled back), compare the new model’s observed post-switch success in that lane against the prior model’s baseline; if it’s measurably worse with enough samples, auto-revert to the prior. Never self-graded — the signal is the outcome ledger’s verifier/outcome receipts. The daemon calls this on its periodic tick. Returns the reverted lanes.

pub async fn concierge_ask(&self, question: &str) -> Result<String, String>

Conversational concierge (Phase F1/F2): answer a free-form question about the user’s models/portfolio, grounded in the observed-usage evidence + the deterministic recommend() candidate menu. The LLM explains — it runs on a local model, is told to answer ONLY from the supplied evidence, and must not invent a model or assert fit (the grounding oracle already decided fit). This is the ModelConcierge “agent”: a thin, constrained generate call over assembled receipts, not a freelancing chat.

pub async fn refresh_catalog(&self) -> Result<usize, String>

Refresh the model catalog from the configured signed source (Phase E1): fetch + verify (detached ed25519 against the pinned key) + cache the verified models. Source is CAR_CATALOG_URL + CAR_CATALOG_PUBKEY (no key → refused). The new models load into the registry at next startup (the registry is immutable at runtime), then surface as recommend() candidates / concierge suggestions. Returns the number of models in the verified catalog.

pub async fn discover_models(&self) -> Result<usize, String>

Auto-discover provider models (Phase E2): query the provider’s /v1/models list and cache previously-unknown chat/reasoning models as TrustTier::Community entries (cloning a curated same-provider schema as a template). Best-effort — no key or no OpenAI provider configured is a no-op, not an error. Discovered models load into the registry at next startup (the registry is immutable at runtime). Returns the total number of cached discovered models. This is what lets the catalog (and the router) pick up new models like a gpt-5.5 without a release.

pub fn lane_defaults(&self) -> LaneDefaults

All configured lane defaults (Phase D1), from the in-memory cache.

pub fn lane_default( &self, project: Option<&str>, use_case: UseCase, ) -> Option<String>

Resolve the default model for (project, use_case), if set. Routing consults this as a strong preference before falling back to adaptive selection. Reads the cache (no disk).

pub fn set_lane_default( &self, project: Option<String>, use_case: UseCase, model_id: &str, ) -> Result<(), String>

Set the default model for (project, use_case) (Phase D1) — the durable target of the concierge’s “set it up” action.

pub fn clear_lane_default( &self, project: Option<&str>, use_case: UseCase, ) -> Result<bool, String>

Clear the default for (project, use_case). Returns whether one existed (used by rollback in D3).

pub async fn user_set_lane_default( &self, use_case: UseCase, model_id: &str, project: Option<String>, ) -> Result<(), String>

User-facing lane-default set (the concierge.set_default WS path): like set_lane_default but serialized under the concierge action lock AND recorded in the action ledger. Without the ledger entry a manual pin would be invisible to the canary, which could then revert it based on a stale ledger view — so this records a SetDefault (with the prior captured) exactly like apply, keeping the ledger and the live default consistent.

pub async fn user_clear_lane_default( &self, use_case: UseCase, project: Option<String>, ) -> Result<bool, String>

User-facing lane-default clear (the concierge.clear_default WS path): serialized + ledgered like user_set_lane_default. Records a ClearDefault so the canary sees the lane is no longer a standing switch (its latest action is the clear, not a SetDefault).

pub fn concierge_actions(&self, limit: usize) -> Vec<ConciergeActionEntry>

The recorded concierge actions (Phase D2), most recent last.

pub async fn apply_concierge( &self, use_case: UseCase, model_id: &str, project: Option<String>, ) -> Result<ConciergeApplyResult, String>

Closed-loop “set it up” (Phase D3): acquire model_id, then set it as the lane default — capturing the prior default so the change is reversible (rollback_lane). Every step is recorded in the action ledger.

Consent: the caller (CarHost) owns the pre-download confirmation — this primitive assumes the user has already agreed to the (possibly multi-GB) download; the ledger entry is the audit record that it happened. Single-writer: lane defaults assume one concierge writer (CarHost); concurrent applys would last-write-wins the JSON (the F3 canary watcher must coordinate before it becomes a 2nd writer).

pub async fn rollback_lane( &self, use_case: UseCase, project: Option<String>, ) -> Result<Option<String>, String>

Revert a lane default to its value before the last apply (Phase D3 rollback). Restores the prior model (or clears the default if there was none), recording the rollback. Returns the restored model id, or None if the default was cleared / nothing to undo.

pub async fn concierge_status(&self, inference_active: bool) -> ConciergeStatus

Assemble the ambient concierge status (Phase C1): per-lane usage + friction from the outcome ledger, the current grounded decision (evaluate_concierge), and per-model health from the profiles. A pull (the UI asks); proactive push stays separate.

pub async fn detect_upgrades(&self) -> Vec<UpgradeFinding>

Detect upgrades combining curated rules with upstream Hub discovery, honoring update preferences (channel/policy) and the TTL cache. Upstream probing only happens on the Latest channel and is offline-safe.

pub fn list_schemas(&self) -> Vec<ModelSchema>

List all known models and their download status (legacy). List all model schemas from the unified registry (full metadata).

pub fn catalog_snapshot(&self) -> Result<CatalogSnapshot, String>

Deterministic immutable catalog view used to bind inference routing to exact model rows. Runtime availability never participates in either row digests or the catalog revision.

pub fn registered_schema(&self, id: &str) -> Option<ModelSchema>

Return one registered schema without refreshing availability.

This is for identity/provenance checks that must reflect signed catalog overrides while remaining local and side-effect free.

pub fn list_models(&self) -> Vec<ModelInfo>

pub fn knows_model(&self, name: &str) -> bool

Whether a caller-supplied model name resolves to a registered schema.

Answers the question generation asks, by the same two routes and in the same order: exact id, then the case-insensitive name lookup. It is deliberately NOT list_models(), which returns the on-device catalog — checking a remote model id against that set reports every cloud model as unknown.

Exists so a caller that fans out to several named models can refuse a typo up front instead of discovering it as a generation error per request. Says nothing about whether the model is currently reachable (credentials, network) — only that the name is one CAR knows.

pub fn model_schema(&self, name: &str) -> Option<&ModelSchema>

The registered schema behind a model name or id, resolved exactly as Self::knows_model resolves it — exact id first, then the case-insensitive name lookup.

Defined together with knows_model so the two cannot drift into disagreeing about which names exist, and it resolves in the SAME order generation does (get(id).or_else(find_by_name(id)), as at the routing sites) — so a caller asking “will these two names reach the same model?” gets the answer that will actually hold at generation time, including find_by_name’s MLX-variant redirect on Apple silicon.

That fidelity is the point, and it is NOT a canonical identity oracle. The lookup is over a HashMap, so if two rows share a display name the one returned is arbitrary — stable within a process, not across restarts. Generation has the same property, so a caller comparing what will run stays correct; a caller needing a stable identity for storage wants the exact id via registered_schema.

pub async fn pull_model(&self, name: &str) -> Result<PathBuf, InferenceError>

Download a model if not already present.

pub async fn pull_model_with_progress( &self, name: &str, sink: &ProgressSink, ) -> Result<PathBuf, InferenceError>

Download a model if not already present, reporting progress to sink and enforcing the acquisition lifecycle (per-model lock, disk preflight, lifecycle events). The CLI and daemon use this to show live progress.

pub async fn adopt_model_into_car( &self, model_id: &str, ) -> Result<InstallReceipt, InferenceError>

Explicitly adopt an already-usable local artifact into CAR ownership. The source path is resolved from the registry; callers cannot nominate an arbitrary deletion target.

pub async fn remove_model_from_car( &self, model_id: &str, ) -> Result<RemoveFromCarResult, InferenceError>

Safely remove only CAR-owned linkage after every Task 3 runtime owner acknowledges release. Shared Hugging Face blobs remain untouched.

pub fn update_prefs(&self) -> UpdatePreferences

Current update preferences. A team-shared project .car/update-prefs.json (found by walking up from cwd) overrides the user ~/.car/update-prefs.json; defaults if neither exists. Loaded on demand — read at onboarding/ upgrade-check frequency, not on the inference hot path.

pub fn set_update_prefs( &self, prefs: &UpdatePreferences, ) -> Result<(), InferenceError>

Persist update preferences to ~/.car/update-prefs.json.

pub fn remove_model(&self, name: &str) -> Result<(), InferenceError>

👎Deprecated:

use async remove_model_from_car for receipt-backed safe removal

Legacy synchronous removal is intentionally disabled because it cannot coordinate active workers, cross-process leases, or receipt ownership. Use Self::remove_model_from_car instead.

pub fn register_model(&mut self, schema: ModelSchema)

Register a model at the public runtime boundary.

The registry normalizes every such schema to Community trust. Project curation is reserved for compiled builtins and signature-verified catalogs inside this crate.

pub fn register_user_model(&mut self, schema: ModelSchema)

Register a model from a user-controlled schema boundary.

pub async fn discover_vllm_mlx_models(&mut self) -> usize

Discover generic MLX models from a running vLLM-MLX server and register them. Returns the number of discovered models added or refreshed in the registry.

pub fn outcome_tracker(&self) -> Arc<RwLock<OutcomeTracker>>

Get outcome tracker for external use (e.g., memgine integration).

pub async fn save_outcomes(&self) -> Result<(), Error>

Persist outcome profiles to disk for cross-session learning (#13). Unconditional (force) save — writes even if nothing changed. Prefer flush_outcomes for shutdown / periodic flushes; the per-call path uses auto_save_outcomes, which debounces.

pub async fn flush_outcomes(&self) -> Result<bool, Error>

Flush outcome profiles to disk iff dirty, ignoring the per-call time debounce. Returns whether a write happened. This is the durable-receipt backstop: the daemon calls it on a periodic timer and on graceful shutdown so the last (sub-OUTCOME_FLUSH_INTERVAL) window of learning is never lost. Cheap when clean (no write).

pub async fn prune_outcome_ledger(&self, max_entries: usize) -> Result<()>

Enforce the outcome-ledger retention bound (privacy + disk). A cheap no-op when under the cap; the daemon calls it periodically.

pub async fn save_key_pool_stats(&self) -> Result<(), Error>

Persist key pool stats to disk.

pub async fn key_pool_stats(&self) -> HashMap<String, Vec<KeyStats>>

Get key pool stats for all endpoints.

pub async fn export_profiles(&self) -> Vec<ModelProfile>

Export model performance profiles for persistence.

pub fn outcome_scoreboard(&self) -> Scoreboard

Fold the durable outcome ledger into the deployment scoreboard — the per-model, priced, OUTCOME-DENOMINATED view (cost-per-success, tokens-per-success, success-rate). Reads the same outcome_ledger.jsonl the tracker flushes to (cross-session, survives restart) and joins per-model catalog prices from the registry so usd_per_success is the honest “cry once” figure. Unpriced models keep a None dollar figure rather than a fabricated one. See crate::scoreboard::Scoreboard.

pub async fn import_profiles(&self, profiles: Vec<ModelProfile>)

Import model performance profiles (from persistence).

pub async fn prepare_speech_runtime(&self) -> Result<PathBuf, InferenceError>

Ensure the managed local speech runtime exists and return its root directory — the same root speech_health reports, on every platform.

Apple Silicon used to short-circuit here: native MLX backends were taken to replace the Python runtime outright, so this only created models_dir and handed that back without ever provisioning the managed runtime. Since #640 the runtime is a live fallback there too (the native backends can’t load every catalogued checkpoint) and speech doctor reports its real state — so a “successful” install contradicted doctor, printed a path doctor never mentions, and pushed the multi-minute venv+pip bootstrap onto the first synthesis (Parslee-ai/car#649). Provision it up front on every platform instead.

The one asymmetry that remains is what a bootstrap failure means. Off Apple Silicon the managed runtime is the only local speech path, so failing to build it fails the call. On Apple Silicon it sits behind working native backends, so a missing uv degrades rather than breaks: the root comes back either way, and callers should report speech_health().runtime.installed rather than read a returned path as proof of success. Either way the returned directory exists — a method called “prepare” leaves the thing prepared (Parslee-ai/car#626).

pub fn set_speech_policy(&mut self, policy: SpeechPolicy)

Override speech routing preferences for the current engine instance.

pub fn set_routing_config(&mut self, config: RoutingConfig)

pub async fn install_curated_speech( &mut self, ) -> Result<Vec<SpeechInstallReport>, InferenceError>

Download the curated local speech model set into the shared Hugging Face cache.

pub fn speech_health(&self) -> SpeechHealthReport

Report speech runtime, model cache, and remote-provider health.

pub async fn model_health(&self) -> ModelHealthReport

Report the current model catalog, configured defaults, capability coverage, and speech runtime/provider health in one place.

pub async fn smoke_test_speech( &self, local: bool, remote: bool, ) -> Result<SpeechSmokeReport, InferenceError>

Run a real speech smoke test through the configured local and/or remote paths.

Trait Implementations

impl InferenceHandle for InferenceEngine

fn generate<'life0, 'async_trait>( &'life0 self, req: GenerateRequest, ) -> Pin<Box<dyn Future<Output = Result<String, InferenceError>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait,

Run a generation request to completion. Same contract as InferenceEngine::generate: caller passes a GenerateRequest (which may carry an explicit model, a routing hint, tools, or a thinking budget), receives the final text or an InferenceError.

fn embed<'life0, 'async_trait>( &'life0 self, req: EmbedRequest, ) -> Pin<Box<dyn Future<Output = Result<Vec<Vec<f32>>, InferenceError>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait,

Encode one or more texts as embedding vectors. Same contract as InferenceEngine::embed: returns one Vec<f32> per input text in the same order.