Struct Executor 

pub struct Executor<B: Backend> { /* private fields */ }

A graph, its weights on one backend, and a plan per bucket built on first use.

Implementations

impl<B: Backend> Executor<B>

pub fn new( backend: B, tensors: &[HostTensor<'_>], graph: Graph, buckets: &Buckets, stage: &str, vocab: usize, types: usize, ) -> Result<Self>

Uploads the weights. vocab and types bound the token ids and question types a batch may hold, and stage names the buckets to use from buckets.

Errors

From Backend::upload.

pub fn backend(&self) -> &B

The backend.

pub fn warm(&self) -> impl Iterator<Item = Bucket> + '_

The buckets with a plan built.

pub fn plans_mut(&mut self) -> impl Iterator<Item = &mut B::Plan> + '_

The plans built so far, to inspect or profile.

pub fn prepare(&mut self, bucket: Bucket) -> Result<usize>

Builds the plan for bucket now rather than on first use.

Errors

From Backend::lower.

pub fn run(&mut self, batch: &Batch<'_>, out: &mut Outputs) -> Result<Bucket>

Runs a batch in the smallest bucket that holds it. Once that bucket’s plan exists and out has grown to the batch size, this allocates nothing.

Errors

Error::Batch for an inconsistent batch, Error::NoBucket for one too big, and anything the backend reports.

Trait Implementations

impl<B: Debug + Backend> Debug for Executor<B>

where B::Weights: Debug, B::Plan: Debug,

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.