Struct carton::carton::Carton

pub struct Carton { /* private fields */ }

Implementations

impl Carton

pub async fn load<P: AsRef<str>>(url_or_path: P, opts: LoadOpts) -> Result<Self>

Load a carton given a url, path, etc and options

pub async fn infer<I, S, T>( &self, tensors: I ) -> Result<HashMap<String, Tensor<RunnerStorage>>>where I: IntoIterator<Item = (S, Tensor<T>)>, String: From<S>, T: TensorStorage,

Infer using a set of inputs. Consider using seal and infer_with_handle in pipelines

pub async fn streaming_infer<'a, I, S, T>( &'a self, tensors: I ) -> impl Stream<Item = Result<HashMap<String, Tensor<RunnerStorage>>>> + 'awhere I: IntoIterator<Item = (S, Tensor<T>)> + 'a, String: From<S>, T: TensorStorage,

Infer using a set of inputs. This method has support for intermediate streaming responses Consider using seal and streaming_infer_with_handle in pipelines

pub async fn seal<T>( &self, tensors: HashMap<String, Tensor<T>> ) -> Result<SealHandle>where T: TensorStorage,

“Seal” a set of inputs that will be used for inference. This lets carton start processing tensors (e.g. moving them to the correct devices) before actually running inference and can lead to more efficient pipelines.

pub async fn infer_with_handle( &self, handle: SealHandle ) -> Result<HashMap<String, Tensor<RunnerStorage>>>

Infer using a handle from seal. This approach can make inference pipelines more efficient vs just using infer

pub async fn pack<T, O, P: AsRef<str>>(path: P, opts: O) -> Result<PathBuf>where T: TensorStorage, O: Into<PackOpts<T>>,

Pack a carton given a path and options. Returns the path of the output file

pub async fn load_unpacked<T, O, P: AsRef<str>>( path: P, pack_opts: O, load_opts: LoadOpts ) -> Result<Self>where T: TensorStorage + 'static, O: Into<PackOpts<T>>,

Pack a carton given a path and options Functionally equivalent to pack followed by load, but implemented in a more optimized way

pub fn get_info(&self) -> &CartonInfoWithExtras<GenericStorage>

Get info for the loaded model

pub async fn get_model_info<P: AsRef<str>>( url_or_path: P ) -> Result<CartonInfoWithExtras<GenericStorage>>

Get info for a model

pub async fn shrink( path: PathBuf, urls: HashMap<String, Vec<String>> ) -> Result<PathBuf>

Shrink a packed carton by storing links to files instead of the files themselves when possible. Takes a path to a packed carton along with a mapping from sha256 to a list of URLs Returns the path to another packed carton

pub async fn alloc_tensor( &self, dtype: DataType, shape: Vec<u64> ) -> Result<Tensor<RunnerStorage>>

Allocate a tensor