Struct Model 

pub struct Model {
    pub path: PathBuf,
    pub name: PathBuf,
    pub quant: usize,
    pub quant_type: Quant,
    pub precision: Precision,
    pub token_chunk_size: usize,
    pub max_batch: usize,
}

Fields

path: PathBuf

Path to the folder containing all models.

name: PathBuf

Name of the model.

quant: usize

Specify layers that needs to be quantized.

quant_type: Quant

Quantization type (Int8 or NF4).

precision: Precision

Precision for intermediate tensors (Fp16 or Fp32).

token_chunk_size: usize

Maximum tokens to be processed in parallel at once.

max_batch: usize

Number of states that are cached on GPU.

Trait Implementations

impl Clone for Model

fn clone(&self) -> Model

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for Model

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for Model

fn default() -> Self

Returns the “default value” for a type.

impl<'de> Deserialize<'de> for Model

where Model: Default,

fn deserialize<__D>(__deserializer: __D) -> Result<Self, __D::Error>

where __D: Deserializer<'de>,

Deserialize this value from the given Serde deserializer.

impl Serialize for Model

fn serialize<__S>(&self, __serializer: __S) -> Result<__S::Ok, __S::Error>

where __S: Serializer,

Serialize this value into the given Serde serializer.

impl ToSchema for Model

fn to_schema(components: &mut Components) -> RefOr<Schema>

Returns a tuple of name and schema or reference to a schema that can be referenced by the name or inlined directly to responses, request bodies or parameters.