Struct Kime 

pub struct Kime { /* private fields */ }

A loaded model on a device. Clones share it, and it can be used from any thread.

Implementations

impl Kime

pub fn builder() -> Builder

Settings for a new engine.

pub fn model_id(&self) -> &str

The model’s name, laya or laya-multilingual for the published checkpoints.

pub fn device(&self) -> String

The device the model runs on, in words.

pub fn max_row_tokens(&self) -> usize

The most tokens one question’s row can hold: the state, the question and its options. Longer states are cut to fit.

pub fn memory(&self) -> Memory

The bytes the model holds on its device, as of the last forward pass.

pub fn count_tokens(&self, req: &Request) -> usize

The tokens a request holds before anything is cut: its state once plus every question with its options. The server refuses a request over its limit with this count, before it is queued.

pub fn decide(&self, req: &Request) -> Result<Response, Error>

Answers one request.

Errors

Error::Invalid for a request that does not validate, Error::TooLong for a question whose options do not fit, and device errors.

pub fn decide_batch(&self, reqs: &[Request]) -> Result<Vec<Response>, Error>

Answers many requests, packing all their questions into as few device batches as fit. Each answer is the same bits it would be alone, whatever else is in the batch.

Errors

As Kime::decide. One bad request fails the whole call.

pub fn decide_batch_timed( &self, reqs: &[Request], ) -> Result<(Vec<Response>, Timing), Error>

Kime::decide_batch, and where the time went.

Errors

As Kime::decide_batch.

pub fn decide_async(&self, req: &Request) -> Decision

Kime::decide on a thread of its own, for async callers. It works with any executor, since it needs nothing from one but a waker.

Trait Implementations

impl Clone for Kime

fn clone(&self) -> Self

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for Kime

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.