Struct Compat 

pub struct Compat { /* private fields */ }

A compat checkpoint ready to run on the CPU.

Implementations

impl Compat

pub fn new(model: &Model, threads: usize) -> Self

Converts the weights of model to f32, on threads threads, which the forward pass uses too.

pub fn from_parts( spec: &LayaSpec, graph: &LayaGraph, t: &Tensors, threads: usize, ) -> Self

Compat::new from the pieces of a model.

pub fn set_threads(&mut self, threads: usize)

Changes the number of threads the forward pass uses. The results do not change with it.

pub fn spec(&self) -> &LayaSpec

The checkpoint’s configuration.

pub fn forward(&self, batch: &[Input<'_>]) -> Vec<Output>

Runs a batch. Sequences are packed end to end, so a batch costs what its tokens cost.

Panics

If a token id is past the vocabulary, a marker is past its sequence, or a qtype is not 0, 1 or 2.

Trait Implementations

impl Debug for Compat

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.