car-inference 0.54.0

Local model inference for CAR — Candle backend with Qwen3 models
Documentation

car-inference

Local and remote model inference for the Common Agent Runtime.

What it does

Provides on-device inference using Candle (Metal on macOS, CUDA on Linux) with Qwen3 models downloaded from HuggingFace on first use. Also supports remote APIs (OpenAI, Anthropic, Google) via the same typed ModelSchema interface. The AdaptiveRouter selects the best model using a filter-score-explore strategy, learning from outcomes over time via OutcomeTracker.

Local-model resource policy

CAR stores its local-model admission preference at $CAR_HOME/model-resource-policy.json (normally ~/.car/model-resource-policy.json). Everyday is the safe default and keeps 60% of system or Apple unified memory available for macOS and other applications. Local-model focused keeps 20% available. Custom records an exact allocation in MB; 0 MB disables new CAR-managed local-model loads without deleting weights or changing the selected model.

Every profile also preserves an emergency reserve of at least 2 GB or 10% of total memory, whichever is larger. A Custom value copied to a smaller machine is clamped only when evaluated and returns a notice; CAR does not silently rewrite the saved preference. This is a CAR admission limit, not an operating-system-enforced RSS cap. Live-memory admission is a separate check at model load time.

Apple Silicon and CPU inference use the system/unified-memory policy. On a discrete CUDA machine, the same policy governs CAR's host-memory admission, while measured VRAM remains a separate weight-fit pool; Apple unified-memory percentages are never applied to CUDA VRAM.

Usage

use car_inference::{InferenceEngine, InferenceConfig, GenerateRequest, GenerateParams};

let engine = InferenceEngine::new(InferenceConfig::default());
let result = engine.generate(GenerateRequest {
    prompt: "Explain quicksort".into(),
    params: GenerateParams::default(),
    ..Default::default()
}).await?;

Codex subscription backend

The builtin catalog row openai/gpt-5.6-sol:high runs one text-only inference turn through the locally installed codex exec. Codex remains the sole holder of the ChatGPT-subscription credential: CAR does not read Codex's auth store, removes OPENAI_API_KEY from the child environment, and requires no OpenAI API key.

let result = engine.generate_tracked(GenerateRequest {
    prompt: "Draft a concise morning-news brief from these notes: ...".into(),
    model: Some("openai/gpt-5.6-sol:high".into()),
    params: GenerateParams {
        max_tokens: 1056,
        ..Default::default()
    },
    ..Default::default()
}).await?;

The :high suffix is part of the catalog identity and maps to Codex model_reasoning_effort="high"; the CLI receives the underlying model name gpt-5.6-sol. The process is ephemeral, uses an empty read-only scratch working directory, ignores user/project rules and config, and explicitly disables shell, multi-agent, app, plugin, code-mode, and web-search features. Requests carrying tools, images/audio/video, multi-turn messages, or a response schema fail as unsupported instead of being silently weakened. Successful results include the turn.completed.usage counts emitted by Codex; CAR never estimates a missing count.

This source is intentionally different from car-external-agents' external:codex path. The external-agent adapter is for autonomous coding work: it grants a workspace, permits Codex tools, exposes sessions and streaming agent events, and can take multiple turns. Reusing it for ordinary inference would give a text-generation request agentic side effects. CodexCli instead lives at the model-provider/catalog seam and accepts exactly one tool-free turn.

Codex exec --json emits completed answer items, not token deltas, so this backend is non-streaming. generate_tracked_stream rejects the source with an explicit unsupported-mode error; use generate or generate_tracked. max_tokens is included as an approximate final-answer limit because the Codex CLI has no per-invocation output-token-cap flag; the returned usage always reports what the provider actually consumed. Set CAR_CODEX_BIN only when the binary is not named codex on PATH.

Apple FoundationModels backend (macOS 26+)

On Apple Silicon Macs running macOS 26 or later with Apple Intelligence provisioned, CAR can route inference to the system LLM through Apple's FoundationModels framework — no model file to download, no API key, no model weights. The OS owns everything.

The integration is a small Swift shim (car-inference/swift/CarFoundationModels.swift) compiled by build.rs and linked into the crate. The framework is weak-linked so the produced binary still loads on pre-26 macOS; a runtime availability check (is_available(), cached for 5 seconds) gates calls.

// Routing happens automatically — request the apple/foundation:default
// model id, or let the AdaptiveRouter pick it for you.
let result = engine.generate(GenerateRequest {
    prompt: "summarize this in one sentence: ...".into(),
    model_id: Some("apple/foundation:default".into()),
    ..Default::default()
}).await?;

The catalog entry is tagged ["builtin", "local", "low_latency", "private"]. The adaptive router scores low_latency AND private together via SYSTEM_LLM_BONUS (0.12) so system-owned models compete fairly with MLX 4B for short fast-turn tasks (autocomplete, summarize, classify) without claiming heavy reasoning workloads they can't serve.

What's wired:

  • Single-shot text generation via generate()
  • Token-by-token streaming via stream() with prefix-diffing on Apple's cumulative snapshots — StreamEvent::Done.text carries the full assembled output, matching Candle/MLX shape.
  • Tool calling via generate_with_tools() — JSON-Schema tool definitions are converted to per-tool DynamicGenerationSchemas so the model sees real schemas. A capture-only Swift Tool records the first invocation and ends the turn; the call comes back as the standard ToolCall shape the remote backends emit and the runtime executes it (CAR's propose/validate/execute contract). At most one tool call per turn — sequential tool use, no parallel calls; the catalog entry claims tool_use but deliberately NOT multi_tool_call, the router-readable "no parallel calls" signal. When a call is captured, text is empty — pre-call assistant prose is discarded.
  • Structured output via generate_structured() — ResponseFormat::JsonSchema maps onto FM's native constrained decoding (respond(to:schema:)). JsonObject has no native FM mode and falls back to instruction injection with a warning.
  • Graceful fallthrough on vision / audio / video — returns InferenceError::UnsupportedMode (streaming included) so the router picks the next candidate instead of silently dropping capabilities.

What isn't:

  • Multimodal input — the public FoundationModels API is text-only.
  • Full JSON-Schema fidelity: the DynamicGenerationSchema conversion covers object/string/number/integer/boolean/array and string enums; typeless nodes (oneOf/anyOf/$ref), union types (["number","null"]), unrecognized types, and non-string enums degrade to permissive string fields (never to an empty object), and numeric enums keep the base type but lose the value constraint. Every degradation is detected on the Rust side and logged via tracing::warn! before crossing the FFI.

Build requirements:

  • Full Xcode (not just Command Line Tools) on the build host — needed for xcrun swiftc and the FoundationModels SDK.
  • macOS 15+ deployment floor (overridable via MACOSX_DEPLOYMENT_TARGET).

Linux and Intel-Mac builds skip the Swift compile entirely; the ModelSource::AppleFoundationModels schema variant still serializes (so registries can describe the model on any platform), but dispatch errors out with UnsupportedMode before reaching the bridge.

Provisioned live lanes

Paid-provider, installed-runtime, and real-weight checks are explicit commands, not ignored tests. Every command requires CAR_INFERENCE_LIVE=1, validates its own credential/runtime/weight prerequisites, exits non-zero when unavailable or below threshold, and writes exactly one JSON receipt to stdout. Run from car-rs/:

# Running vLLM-MLX: health + discovery + registration + generation
CAR_INFERENCE_LIVE=1 VLLM_MLX_ENDPOINT=http://127.0.0.1:8000 \
  cargo run -p car-inference --example provisioned_live -- vllm

# Paid multimodal/tool contracts (provider: gemini, openai, anthropic)
CAR_INFERENCE_LIVE=1 GOOGLE_API_KEY=... \
  cargo run -p car-inference --example provisioned_live -- provider-vision gemini
CAR_INFERENCE_LIVE=1 OPENAI_API_KEY=... \
  cargo run -p car-inference --example provisioned_live -- provider-tool openai

# Local or ElevenLabs speech synthesis-to-transcription roundtrip
CAR_INFERENCE_LIVE=1 \
  cargo run -p car-inference --example provisioned_live -- speech local
CAR_INFERENCE_LIVE=1 ELEVENLABS_API_KEY=... \
  cargo run -p car-inference --example provisioned_live -- speech elevenlabs

# Anthropic prompt-cache write/read, real local usage, and long MLX context
CAR_INFERENCE_LIVE=1 ANTHROPIC_API_KEY=... \
  cargo run -p car-inference --example provisioned_live -- anthropic-cache
CAR_INFERENCE_LIVE=1 \
  cargo run -p car-inference --example provisioned_live -- local-usage
CAR_INFERENCE_LIVE=1 \
  cargo run -p car-inference --example provisioned_live -- long-context

# Gemma 4 real-weight cases
for case in chat tool-generation conclude-after-tool prefix-reuse long-reference load-prefill; do
  CAR_INFERENCE_LIVE=1 cargo run -p car-inference \
    --example provisioned_live -- gemma4 "$case" || exit 1
done

The local lanes refuse to download implicitly: provision the named model first with car models pull. Model-quality comparisons beyond these fixed contract thresholds belong in car-bench.

Crate features

  • metal -- Apple Silicon GPU acceleration
  • cuda -- NVIDIA GPU acceleration
  • ast -- AST-aware code generation via car-ast

Part of CAR -- see the main repo for full documentation.