Struct Kime 

pub struct Kime { /* private fields */ }

A loaded model on a device. Clones share it, and it can be used from any thread.

Implementations

impl Kime

pub fn builder() -> Builder

Settings for a new engine.

pub fn model_id(&self) -> &str

The model’s name, laya or laya-multilingual for the published checkpoints.

pub fn device(&self) -> String

The device the model runs on, in words.

pub fn max_row_tokens(&self) -> usize

The most tokens one question’s row can hold: the state, the question and its options. Longer states are cut to fit.

pub fn memory(&self) -> Memory

The bytes the model holds on its device, as of the last forward pass.

pub fn cache_stats(&self) -> CacheStats

The answer cache’s counts, all 0 when it is off.

pub fn cached(&self, req: &Request) -> Option<Response>

The answer to req when the answer cache holds every one of its questions, found without the device lock, so the server can answer it without queueing it. None otherwise, and then nothing is counted, since the request goes on to Kime::decide_batch.

pub fn count_tokens(&self, req: &Request) -> usize

The tokens a request holds before anything is cut: its state once plus every question with its options. The server refuses a request over its limit with this count, before it is queued.

pub fn decide(&self, req: &Request) -> Result<Response, Error>

Answers one request.

Errors

Error::Invalid for a request that does not validate, Error::TooLong for a question whose options do not fit, and device errors.

pub fn decide_batch(&self, reqs: &[Request]) -> Result<Vec<Response>, Error>

Answers many requests, packing all their questions into as few device batches as fit. Each answer is the same bits it would be alone, whatever else is in the batch.

Errors

As Kime::decide. One bad request fails the whole call.

pub fn decide_batch_timed( &self, reqs: &[Request], ) -> Result<(Vec<Response>, Timing), Error>

Kime::decide_batch, and where the time went.

Errors

As Kime::decide_batch.

pub fn embed( &self, texts: &[&str], max_length: usize, ) -> Result<Vec<Vec<f32>>, Error>

Each text’s encoder output mean pooled over its tokens, as Laya’s embed_fn_from_agent computes it: [CLS], the text cut to max_length tokens with the specials, [SEP]. It runs no decision head. Each row is as wide as the encoder.

Errors

Device errors, and Error::Unsupported on a backend without the pooled graph or in INT8, which puts some rows far from Laya’s.

pub fn decide_async(&self, req: &Request) -> Decision

Kime::decide on a thread of its own, for async callers. It works with any executor, since it needs nothing from one but a waker.

Trait Implementations

impl Clone for Kime

fn clone(&self) -> Self

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for Kime

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.