pub struct Executor<B: Backend> { /* private fields */ }Expand description
A graph, its weights on one backend, and a plan per bucket built on first use. More graphs
can share the weights, see Executor::add_graph.
Implementations§
Source§impl<B: Backend> Executor<B>
impl<B: Backend> Executor<B>
Sourcepub fn new(
backend: B,
tensors: &[HostTensor<'_>],
graph: Graph,
buckets: &Buckets,
stage: &str,
vocab: usize,
types: usize,
) -> Result<Self>
pub fn new( backend: B, tensors: &[HostTensor<'_>], graph: Graph, buckets: &Buckets, stage: &str, vocab: usize, types: usize, ) -> Result<Self>
Uploads the weights. vocab and types bound the token ids and question types a batch may
hold, and stage names the buckets to use from buckets.
§Errors
From Backend::upload.
Sourcepub fn add_graph(
&mut self,
graph: Graph,
buckets: &Buckets,
stage: &str,
) -> usize
pub fn add_graph( &mut self, graph: Graph, buckets: &Buckets, stage: &str, ) -> usize
Adds a graph that runs on the weights already uploaded, in the buckets of stage, and
returns the lane to pass to Executor::run_lane. The graph may only read weights the
first graph reads, the way the first graph reads them, such as the encoder alone.
Sourcepub fn memory(&self) -> (usize, usize)
pub fn memory(&self) -> (usize, usize)
Bytes on the device: the weights, and the plans built so far.
Sourcepub fn plans_mut(&mut self) -> impl Iterator<Item = &mut B::Plan> + '_
pub fn plans_mut(&mut self) -> impl Iterator<Item = &mut B::Plan> + '_
The plans built so far, to inspect or profile.
Sourcepub fn run(&mut self, batch: &Batch<'_>, out: &mut Outputs) -> Result<Bucket>
pub fn run(&mut self, batch: &Batch<'_>, out: &mut Outputs) -> Result<Bucket>
Runs a batch in the smallest bucket that holds it. Once that bucket’s plan exists and out
has grown to the batch size, this allocates nothing.
§Errors
Error::Batch for an inconsistent batch, Error::NoBucket for one too big, and
anything the backend reports.
Sourcepub fn run_lane(
&mut self,
lane: usize,
batch: &Batch<'_>,
out: &mut Outputs,
) -> Result<Bucket>
pub fn run_lane( &mut self, lane: usize, batch: &Batch<'_>, out: &mut Outputs, ) -> Result<Bucket>
Executor::run on the graph lane from Executor::add_graph, 0 being the first.
§Errors
As Executor::run.