Struct WorkerOffload 

pub struct WorkerOffload { /* private fields */ }

Installed on the daemon’s engine to route on-device generation through a persistent, auto-respawning worker child. Cheap to clone (shares the child).

Implementations

impl WorkerOffload

pub fn new() -> Result<Self>

Offload to a fresh car-server --mlx-worker child of the current executable. Fails only if the current exe path can’t be resolved.

pub fn with_command(program: impl Into<OsString>, args: Vec<OsString>) -> Self

Construct with an explicit worker command. Used by tests to point at a stub worker (a script that speaks the same line protocol) so the spawn/exchange/respawn logic is exercised without loading a real model.

Trait Implementations

impl Clone for WorkerOffload

fn clone(&self) -> Self

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl LocalGenerationOffload for WorkerOffload

fn generate<'life0, 'async_trait>( &'life0 self, _request: GenerateRequest, ) -> Pin<Box<dyn Future<Output = Result<InferenceResult, InferenceError>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait,

Run a non-streaming generation to completion in the worker.

fn stream<'life0, 'async_trait>( &'life0 self, _request: GenerateRequest, ) -> Pin<Box<dyn Future<Output = Result<Receiver<StreamEvent>, InferenceError>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait,

Run a streaming generation in the worker, returning a receiver of the same StreamEvents the in-process path emits. The channel closes when the worker finishes (or dies); a mid-stream worker death is observed as the sender dropping, exactly like any other stream end.

fn generate_admitted<'life0, 'async_trait>( &'life0 self, request: GenerateRequest, admission: LocalWorkerAdmission, ) -> Pin<Box<dyn Future<Output = Result<LocalOffloadResult, InferenceError>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait,

Admission-aware extension. The default preserves source compatibility for external legacy implementers and treats their result as transient; only implementations that explicitly override this method may report retained worker/process weights.

fn stream_admitted<'life0, 'async_trait>( &'life0 self, request: GenerateRequest, admission: LocalWorkerAdmission, ) -> Pin<Box<dyn Future<Output = Result<LocalOffloadStream, InferenceError>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait,

fn refresh_resource_policy(&self, generation: u64)

fn resident_allocation_id(&self, model_id: &str) -> Option<String>

Exact process-generation allocation owner for parent-side residency publication. Legacy implementations may omit it and remain transient.

fn resident_models<'life0, 'async_trait>( &'life0 self, ) -> Pin<Box<dyn Future<Output = Vec<String>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait,

fn release_model<'life0, 'life1, 'async_trait>( &'life0 self, model_id: &'life1 str, ) -> Pin<Box<dyn Future<Output = Result<bool, InferenceError>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait,

Release worker-owned residency for one model.

fn release_idle_models<'life0, 'async_trait>( &'life0 self, idle_for: Duration, ) -> Pin<Box<dyn Future<Output = Vec<String>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait,

Release worker-owned model residents whose last completed use is at least idle_for old. Legacy offloads do not track residency age.

fn terminate_inference<'life0, 'life1, 'async_trait>( &'life0 self, inference_id: &'life1 str, ) -> Pin<Box<dyn Future<Output = InferenceTerminationAck> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait,

Terminate the exact isolated process currently executing this ID. The compatibility default is intentionally conservative: only an exact kill-and-wait implementation may report confirmed termination.