Struct LocalRunnerClient 

pub struct LocalRunnerClient<R: ModelRunner> { /* private fields */ }

Wrap any ModelRunner (including MockRunner) as an InferenceClient. Single-runner concurrency is bounded by the internal mutex; production setups should use the EngineCoreActor from atomr-infer-runtime instead.

Implementations

impl<R: ModelRunner + 'static> LocalRunnerClient<R>

pub fn new(runner: R, provider: Provider) -> Self

pub fn from_arc(runner: Arc<Mutex<R>>, provider: Provider) -> Self

Trait Implementations

impl<R: ModelRunner + 'static> InferenceClient for LocalRunnerClient<R>

fn provider(&self) -> Provider

fn run<'life0, 'async_trait>( &'life0 self, batch: ExecuteBatch, ) -> Pin<Box<dyn Future<Output = Result<TurnResult>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait,