car-inference 0.58.0

Local model inference for CAR — Candle backend with Qwen3 models
Documentation

car-inference

Local and remote model inference for the Common Agent Runtime.

What it does

Provides on-device inference using Candle (Metal on macOS, CUDA on Linux) with Qwen3 models downloaded from HuggingFace on first use. Also supports remote APIs (OpenAI, Anthropic, Google) via the same typed ModelSchema interface. The AdaptiveRouter selects the best model using a filter-score-explore strategy, learning from outcomes over time via OutcomeTracker.

Local-model resource policy

CAR stores its local-model admission preference at $CAR_HOME/model-resource-policy.json (normally ~/.car/model-resource-policy.json). Everyday is the safe default and keeps 60% of system or Apple unified memory available for macOS and other applications. Local-model focused keeps 20% available. Custom records an exact allocation in MB; 0 MB disables new CAR-managed local-model loads without deleting weights or changing the selected model.

Every profile also preserves an emergency reserve of at least 2 GB or 10% of total memory, whichever is larger. A Custom value copied to a smaller machine is clamped only when evaluated and returns a notice; CAR does not silently rewrite the saved preference. This is a CAR admission limit, not an operating-system-enforced RSS cap. Live-memory admission is a separate check at model load time.

Apple Silicon and CPU inference use the system/unified-memory policy. On a discrete CUDA machine, the same policy governs CAR's host-memory admission, while measured VRAM remains a separate weight-fit pool; Apple unified-memory percentages are never applied to CUDA VRAM.

Usage

use car_inference::{InferenceEngine, InferenceConfig, GenerateRequest, GenerateParams};

let engine = InferenceEngine::new(InferenceConfig::default());
let result = engine.generate(GenerateRequest {
    prompt: "Explain quicksort".into(),
    params: GenerateParams::default(),
    ..Default::default()
}).await?;

Codex subscription backend

The builtin catalog row openai/gpt-5.6-sol:high runs one tool-free inference turn through the locally installed codex exec. Codex remains the sole holder of the ChatGPT-subscription credential: CAR does not read Codex's auth store, removes OPENAI_API_KEY from the child environment, and requires no OpenAI API key.

let result = engine.generate_tracked(GenerateRequest {
    prompt: "Draft a concise morning-news brief from these notes: ...".into(),
    model: Some("openai/gpt-5.6-sol:high".into()),
    params: GenerateParams {
        max_tokens: 1056,
        ..Default::default()
    },
    ..Default::default()
}).await?;

The :high suffix is part of the catalog identity and maps to Codex model_reasoning_effort="high"; the CLI receives the underlying model name gpt-5.6-sol. The process is ephemeral, uses an empty read-only scratch working directory, ignores user/project rules and config, and explicitly disables shell, multi-agent, app, plugin, code-mode, and web-search features. Inline images are decoded into that scratch directory and passed with --image in conversation order, earlier turns first and the current message last; each message in the encoded conversation lists the 1-based positions of the images it carried, and an image over 32 MiB is refused before it is written. Video, audio, and remote image URL attachments are not sent: the reply names the model, says which attachment it cannot process, and still answers the text. Conversation history and CAR tool proposals are supported; response schemas and streaming are rejected. Successful results include the turn.completed.usage counts emitted by Codex; CAR never estimates a missing count.

This source is intentionally different from car-external-agents' external:codex path. The external-agent adapter is for autonomous coding work: it grants a workspace, permits Codex tools, exposes sessions and streaming agent events, and can take multiple turns. Reusing it for ordinary inference would give a text-generation request agentic side effects. CodexCli instead lives at the model-provider/catalog seam and runs one codex exec invocation per request, with CAR tool calls returned as proposals.

Codex exec --json emits completed answer items, not token deltas, so this backend is non-streaming. generate_tracked_stream rejects the source with an explicit unsupported-mode error; use generate or generate_tracked. max_tokens is included as an approximate final-answer limit because the Codex CLI has no per-invocation output-token-cap flag; the returned usage always reports what the provider actually consumed. Set CAR_CODEX_BIN only when the binary is not named codex on PATH.

A definite "not logged in" from that binary makes the source unavailable: is_available() asks codex login status (local, OPENAI_API_KEY removed as for a real run). The first answer per binary is asked on a background thread (three-second cap), and until it arrives the binary check alone decides; after that the answer is reused for 30 seconds and then refreshed on a background thread, except an expired "not logged in", which is checked again on the spot (same three-second cap) so a Codex signed in since shows up on that read. A definite "Not logged in" makes the row unavailable; an answer CAR can't read keeps the binary-only verdict, so a reworded status line never hides a working CLI.

This check runs on passive catalog and startup paths on purpose. It asks Codex's own status command and never touches CAR's secret store, so the demand-driven Keychain rule (passive paths make zero CAR secret-store operations) still holds. It is bounded (three seconds, cached 30), and it is what keeps a signed-out Codex from being listed and routed as available.

The login check runs on macOS and Linux only. On Windows the binary check alone decides, as it did before: std has no non-blocking anonymous pipes there, so a reader a leftover child kept blocked could not be stopped.

Apple FoundationModels backend (macOS 26+)

On Apple Silicon Macs running macOS 26 or later with Apple Intelligence provisioned, CAR can route inference to the system LLM through Apple's FoundationModels framework — no model file to download, no API key, no model weights. The OS owns everything.

The integration is a small Swift shim (car-inference/swift/CarFoundationModels.swift) compiled by build.rs and linked into the crate. The framework is weak-linked so the produced binary still loads on pre-26 macOS; a runtime availability check (is_available(), cached for 5 seconds) gates calls.

// Routing happens automatically — request the apple/foundation:default
// model id, or let the AdaptiveRouter pick it for you.
let result = engine.generate(GenerateRequest {
    prompt: "summarize this in one sentence: ...".into(),
    model_id: Some("apple/foundation:default".into()),
    ..Default::default()
}).await?;

The catalog entry is tagged ["builtin", "local", "low_latency", "private"]. The adaptive router scores low_latency AND private together via SYSTEM_LLM_BONUS (0.12) so system-owned models compete fairly with MLX 4B for short fast-turn tasks (autocomplete, summarize, classify) without claiming heavy reasoning workloads they can't serve.

What's wired:

  • Single-shot text generation via generate()
  • Token-by-token streaming via stream() with prefix-diffing on Apple's cumulative snapshots — StreamEvent::Done.text carries the full assembled output, matching Candle/MLX shape. A JsonSchema stream runs native constrained decoding instead and arrives as one TextDelta + Done; a stream that carries images is not routed here (the stream path takes text only).
  • Tool calling via generate_with_tools() — JSON-Schema tool definitions are converted to per-tool DynamicGenerationSchemas so the model sees real schemas. A capture-only Swift Tool records the first invocation and ends the turn; the call comes back as the standard ToolCall shape the remote backends emit and the runtime executes it (CAR's propose/validate/execute contract). At most one tool call per turn — sequential tool use, no parallel calls; the catalog entry claims tool_use but deliberately NOT multi_tool_call, the router-readable "no parallel calls" signal. When a call is captured, text is empty — pre-call assistant prose is discarded. A required property that allows null is sent as optional, and its null is filled back into the arguments, against the union branch the generated value satisfies, when the model leaves it out.
  • Structured output via generate_structured() — ResponseFormat::JsonSchema maps onto FM's native constrained decoding (respond(to:schema:)) on both generate and stream. A required property that allows null is sent as optional, and its null is filled back into the output, against the union branch the generated value satisfies, when the model leaves it out. JsonObject has no native FM mode and falls back to instruction injection with a warning, on the text, image and stream paths. When tools are present, response_format is ignored with a warning — one turn is either tool-enabled or schema-constrained.
  • Image input via generate_with_images() (macOS 27+): inline base64 ImageBase64 blocks in the request's images field, non-streaming.
  • Refusals instead of silent drops — InferenceError::UnsupportedMode from backend foundation-models, so the router picks the next candidate: mode multimodal-content for audio/video, images inside messages, images together with tools, a URL / text / audio / video block in images, and any image on the stream path; mode structured-output for images together with a JsonSchema.

What isn't:

  • Image input beyond the non-streaming path: images are accepted only as inline base64 in images, only without streaming, and only on macOS 27+. Audio and video input are not supported.
  • Full JSON-Schema fidelity: the DynamicGenerationSchema conversion covers object/string/number/integer/boolean/array, string enums, const, and unions (anyOf, oneOf, type arrays such as ["number","null"]). What still degrades: allOf, not and $ref (ignored beside a type, a permissive string without one), typeless nodes, non-string enums (the base type is kept, the values are not enforced), a oneOf whose branches can overlap (sent as anyOf, so exclusivity is not enforced), null outside an object property, and array without items (elements become strings). Nothing degrades to an empty object. Every degradation is detected on the Rust side and logged via tracing::warn! before crossing the FFI.

Build requirements:

  • Full Xcode (not just Command Line Tools) on the build host — needed for xcrun swiftc and the FoundationModels SDK.
  • macOS 15+ deployment floor (overridable via MACOSX_DEPLOYMENT_TARGET).

Linux and Intel-Mac builds skip the Swift compile entirely; the ModelSource::AppleFoundationModels schema variant still serializes (so registries can describe the model on any platform), but dispatch errors out with UnsupportedMode before reaching the bridge.

Provisioned live lanes

Paid-provider, installed-runtime, and real-weight checks are explicit commands, not ignored tests. Every command requires CAR_INFERENCE_LIVE=1, validates its own credential/runtime/weight prerequisites, exits non-zero when unavailable or below threshold, and writes exactly one JSON receipt to stdout. Run from car-rs/:

# Running vLLM-MLX: health + discovery + registration + generation
CAR_INFERENCE_LIVE=1 VLLM_MLX_ENDPOINT=http://127.0.0.1:8000 \
  cargo run -p car-inference --example provisioned_live -- vllm

# Paid multimodal/tool contracts (provider: gemini, openai, anthropic)
CAR_INFERENCE_LIVE=1 GOOGLE_API_KEY=... \
  cargo run -p car-inference --example provisioned_live -- provider-vision gemini
CAR_INFERENCE_LIVE=1 OPENAI_API_KEY=... \
  cargo run -p car-inference --example provisioned_live -- provider-tool openai

# Local or ElevenLabs speech synthesis-to-transcription roundtrip
CAR_INFERENCE_LIVE=1 \
  cargo run -p car-inference --example provisioned_live -- speech local
CAR_INFERENCE_LIVE=1 ELEVENLABS_API_KEY=... \
  cargo run -p car-inference --example provisioned_live -- speech elevenlabs

# Anthropic prompt-cache write/read, real local usage, and long MLX context
CAR_INFERENCE_LIVE=1 ANTHROPIC_API_KEY=... \
  cargo run -p car-inference --example provisioned_live -- anthropic-cache
CAR_INFERENCE_LIVE=1 \
  cargo run -p car-inference --example provisioned_live -- local-usage
CAR_INFERENCE_LIVE=1 \
  cargo run -p car-inference --example provisioned_live -- long-context

# Gemma 4 real-weight cases
for case in chat tool-generation conclude-after-tool prefix-reuse long-reference load-prefill; do
  CAR_INFERENCE_LIVE=1 cargo run -p car-inference \
    --example provisioned_live -- gemma4 "$case" || exit 1
done

The local lanes refuse to download implicitly: provision the named model first with car models pull. Model-quality comparisons beyond these fixed contract thresholds belong in car-bench.

Crate features

  • metal -- Apple Silicon GPU acceleration
  • cuda -- NVIDIA GPU acceleration
  • ast -- AST-aware code generation via car-ast

Part of CAR -- see the main repo for full documentation.