Module offload 

On-device generation offload — process isolation for local inference.

Closes the second half of [Parslee-ai/car-releases#74]. A single heavy on-device generation (a large MLX Qwen prompt at a high max_tokens) can abort the process from the Metal/MLX C++ side — an allocation/OOM abort or a C++ exception crossing the FFI boundary below every Rust catch_unwind guard (crate::InferenceEngine’s catch_mlx, handle_infer’s catch_unwind). In the shared car-server daemon that takes down inference for every connected client at once.

The fix is to run on-device generation in a separate worker process the daemon owns. When a LocalGenerationOffload is installed (via set_local_offload), crate::InferenceEngine’s on-device branch hands the fully-resolved request to it instead of running the Metal decode loop in-process. A Metal abort then kills only the worker: the offload call returns an InferenceError, the daemon fails that one RPC gracefully and stays up, and the next call respawns the worker.

The slot is process-wide and defaults to None, so in-process consumers (NAPI/PyO3/CLI, the worker itself) run on-device generation directly and are completely unaffected — only a host that explicitly installs an offloader (the daemon) gets the subprocess boundary.

Structs

ControlledTerminationToken

Request-scoped evidence that the exact isolated backend was killed and reaped. It is deliberately monotonic: only a confirmed kill + wait may set it, and every adaptive/retry boundary can cheaply observe it afterward.

LocalOffloadResult

LocalOffloadStream

LocalWorkerAdmission

LocalWorkerResidency

RemoteDeadline

One armed caller deadline: when it expires, and the caller’s own number (timeout_ms) so a termination can NAME the deadline that was applied instead of reporting an anonymous transport ceiling.

Enums

InferenceTerminationAck

Evidence returned by an isolated generation backend after a termination request. Dropping a body/stream/future is not evidence and must remain Unconfirmed.

Traits

LocalGenerationOffload

Handler that runs a fully-resolved on-device GenerateRequest somewhere isolated from the calling process (the daemon’s car-server --mlx-worker child). Implementations must run the request against a real in-process engine and return the same InferenceResult / StreamEvent stream the caller would have produced itself — the boundary is transparent except that a native abort surfaces as an Err/dropped stream instead of a crash.

Functions

clear_remote_deadline

Drop the deadline entry once the inference reaches ANY terminal state. Idempotent; the registry must never outlive its inference or the next request to reuse an ID would inherit a stale deadline.

current_controlled_termination_token

Snapshot the request’s controlled-termination token. Spawned backend tasks must clone this explicitly because Tokio task-local values are not inherited.

current_inference_control_id

Snapshot the lifecycle ID in the current inference task, if one was scoped.

current_local_offload

Snapshot the currently installed offloader (if any). Cheap Arc clone.

current_remote_deadline

The armed deadline for the CURRENT inference task, if any. Reads the task-local lifecycle ID and then the registry, so a call outside a scoped inference (embeddings, health probes) sees None and keeps its defaults.

ensure_not_controlled_terminated

Stop an adaptive retry/fallback boundary after exact backend termination.

is_offload_worker

Whether this process is an on-device inference worker. Set by the daemon on the car-server --mlx-worker child it spawns. A worker runs generation in-process (the real Metal path), never offloads, and disables outcome persistence so it can’t race the parent daemon writing the shared ~/.car profile/ledger files.

scope_inference_control_id

Scope a server-minted lifecycle ID to one inference future without adding a serializable field to GenerateRequest. Direct local-offload awaits inherit this value; spawned tasks do not inherit it.

set_local_offload

Install a process-wide on-device generation offloader. Pass None to clear the slot. Re-registering overwrites any previous offloader.

set_remote_deadline

Arm (or re-arm) the caller’s deadline for one inference. timeout is measured from now, matching infer.deadline’s timeout_ms semantics.