Struct Model 

pub struct Model { /* private fields */ }

A loaded model plus its metadata and runtime statistics: what the server serves.

Build one with Model::custom, or with furnace-burn for a Burn model. Requests run through Model::infer / Model::infer_many (named tensors), or the flat Model::predict family for a model of one f32 tensor in and one out. All of them block; call them off the async runtime.

Implementations

impl Model

pub fn custom(model: impl TensorModel + 'static) -> Self

Serve your own model (see TensorModel)

pub fn new( inner: Box<dyn TensorModel>, model_type: &'static str, model_size_bytes: u64, ) -> Self

Wrap a model; model_type and model_size_bytes are reported by /model/info

pub fn batches_named(&self) -> bool

Whether concurrent requests of named tensors can be joined into one batch: every input has a fixed per-item shape. (Simple models batch on the flat path instead.)

pub fn is_simple(&self) -> bool

Whether the model takes one f32 tensor and returns one, with fixed per-item shapes. Such models serve /predict and batch concurrent requests together; the others take named tensors through Model::infer.

pub fn inputs(&self) -> &[IoSpec]

The inputs, in the order Model::infer takes them

pub fn outputs(&self) -> &[IoSpec]

The outputs, in the order Model::infer returns them

pub fn infer(&self, inputs: Vec<HostTensor>) -> Result<Vec<HostTensor>>

Run one request of named tensors: inputs in the order of Model::inputs, each with the batch as its first dimension. Returns the outputs in the order of Model::outputs. Blocking, like Model::predict_batch.

pub fn infer_many( &self, requests: Vec<Vec<HostTensor>>, ) -> Vec<Result<Vec<HostTensor>>>

Several requests of named tensors as one forward pass, joined along the batch dimension, the outputs split back. Each request is checked on its own, so an invalid one gets its own error and the rest still run. Blocking, like Model::predict_batch.

pub fn predict(&self, input: Vec<f32>) -> Result<Vec<f32>>

Run one item of a simple model (see Model::is_simple): its flattened input in, its flattened output back. Blocking, like Model::predict_batch.

pub fn predict_batch(&self, inputs: Vec<Vec<f32>>) -> Result<Vec<Vec<f32>>>

Run inference on a batch of flattened inputs.

This is CPU/GPU-bound and blocking; call it from a blocking context (e.g. tokio::task::spawn_blocking) inside async code.

pub fn predict_flat( &self, input: Vec<f32>, batch_size: usize, ) -> Result<Vec<f32>>

Run inference on batch_size items stored back to back in one row-major buffer.

Returns the outputs the same way: batch_size * output_size values. This avoids per-item allocations and copies, so it is the fastest entry point for large inputs. Blocking, like Model::predict_batch.

pub fn predict_many(&self, requests: Vec<FlatInput>) -> Vec<Result<Vec<f32>>>

Run several independent requests as one forward pass and split the outputs back.

Each request is validated on its own, so an invalid one gets its own error and the rest still run. Results come back in request order, each holding that request’s batch_size * output_size values. The requests together must not exceed MAX_BATCH_SIZE items. Statistics count every request, plus one batch per forward pass. Blocking, like Model::predict_batch.

pub fn adopt_stats(&mut self, previous: &Model)

Continue previous’s statistics, so a reloaded model’s counters do not restart at zero. The two share them from now on: requests still running on previous count as well.

pub fn warm_up(&self) -> Result<Option<Duration>>

Run one all-zero input through the model without touching the statistics, so the first real request does not pay for lazy initialization (weight transforms, buffer pools, page faults on the memory-mapped weights). Returns how long it took.

A model with an input whose shape is only fixed per request cannot be warmed up without guessing it, and is skipped (Ok(None)).

pub fn get_info(&self) -> &ModelInfo

Name, shapes, backend and the rest of what GET /model/info reports

pub fn get_stats(&self) -> ModelStats

A snapshot of the request counters

pub fn validate_input_shape(&self, input: &[f32]) -> Result<()>

Whether input has one item’s worth of values for a simple model’s input

pub fn get_backend_info(&self) -> String

The backend’s name, as in ModelInfo::backend

pub fn model_type(&self) -> &'static str

How the model was defined, as in ModelInfo::model_type

Trait Implementations

impl Debug for Model

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.