Struct llama_core::graph::Graph

pub struct Graph {
    pub created: Duration,
    pub metadata: Metadata,
    /* private fields */
}

Wrapper of the wasmedge_wasi_nn::Graph struct

Fields

created: Duration

metadata: Metadata

Implementations

impl Graph

pub fn new(metadata: &Metadata) -> Result<Self, LlamaCoreError>

Create a new computation graph from the given metadata.

pub fn name(&self) -> &str

Get the name of the model

pub fn alias(&self) -> &str

Get the alias of the model

pub fn prompt_template(&self) -> PromptTemplateType

Get the prompt template type

pub fn update_metadata(&mut self) -> Result<(), LlamaCoreError>

Update metadata

pub fn set_input<T: Sized>( &mut self, index: usize, tensor_type: TensorType, dimensions: &[usize], data: impl AsRef<[T]>, ) -> Result<(), WasiNnError>

Set input uses the data, not only u8, but also f32, i32, etc.

pub fn compute(&mut self) -> Result<(), WasiNnError>

Compute the inference on the given inputs.

pub fn compute_single(&mut self) -> Result<(), WasiNnError>

Compute the inference on the given inputs.

Note that this method is used for the stream mode. It generates one token at a time.

pub fn get_output<T: Sized>( &self, index: usize, out_buffer: &mut [T], ) -> Result<usize, WasiNnError>

Copy output tensor to out_buffer, return the output’s size in bytes.

pub fn get_output_single<T: Sized>( &self, index: usize, out_buffer: &mut [T], ) -> Result<usize, WasiNnError>

Copy output tensor to out_buffer, return the output’s size in bytes.

Note that this method is used for the stream mode. It returns one token at a time.

pub fn finish_single(&mut self) -> Result<(), WasiNnError>

Clear the computation context.

Note that this method is used for the stream mode. It clears the context after the stream mode is finished.

Trait Implementations

impl Debug for Graph

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.