Struct MultiModelConfig

#[non_exhaustive]
pub struct MultiModelConfig {
    pub model_cache_setting: Option<ModelCacheSetting>,
}

Specifies additional configuration for hosting multi-model endpoints.

Fields (Non-exhaustive)

Non-exhaustive structs could have additional fields added in future. Therefore, non-exhaustive structs cannot be constructed in external crates using the traditional Struct { .. } syntax; cannot be matched against without a wildcard ..; and struct update syntax will not work.

model_cache_setting: Option<ModelCacheSetting>

Whether to cache models for a multi-model endpoint. By default, multi-model endpoints cache models so that a model does not have to be loaded into memory each time it is invoked. Some use cases do not benefit from model caching. For example, if an endpoint hosts a large number of models that are each invoked infrequently, the endpoint might perform better if you disable model caching. To disable model caching, set the value of this parameter to Disabled.

Implementations

impl MultiModelConfig

pub fn model_cache_setting(&self) -> Option<&ModelCacheSetting>

Whether to cache models for a multi-model endpoint. By default, multi-model endpoints cache models so that a model does not have to be loaded into memory each time it is invoked. Some use cases do not benefit from model caching. For example, if an endpoint hosts a large number of models that are each invoked infrequently, the endpoint might perform better if you disable model caching. To disable model caching, set the value of this parameter to Disabled.

impl MultiModelConfig

pub fn builder() -> MultiModelConfigBuilder

Creates a new builder-style object to manufacture MultiModelConfig.

Trait Implementations

impl Clone for MultiModelConfig

fn clone(&self) -> MultiModelConfig

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for MultiModelConfig

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl PartialEq for MultiModelConfig

fn eq(&self, other: &MultiModelConfig) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl StructuralPartialEq for MultiModelConfig