Enum DecodePrecision 

pub enum DecodePrecision {
    Model,
    Fp16,
}

Decoder-wide numeric precision for the session’s decode graph.

This is a generic, model-agnostic knob selected via SessionBuilder::decode_precision. The default, DecodePrecision::Model, runs the graph exactly as authored, so default runtime behaviour is byte-identical to a build with no precision knob at all.

DecodePrecision::Fp16 requests a whole-decoder fp32→fp16 rewrite (see [fp16_decode]). It only takes effect on a GPU device and only for an fp32-activation int4/block-32 quantized decoder (fp32-scale MatMulNBits); for every other model — including native fp16-activation models — it is a strict no-op, leaving the graph bit-identical.

Variants

Model

Run the decoder at the precision authored in the model graph (default).

Fp16

Cast an fp32-activation int4/block-32 decoder to a fully fp16 graph so a GPU backend runs it through the fp16-fused decode kernels. No-op unless the session targets a GPU and the graph is fp32-activation quantized.

Trait Implementations

impl Clone for DecodePrecision

fn clone(&self) -> DecodePrecision

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Copy for DecodePrecision

impl Debug for DecodePrecision

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for DecodePrecision

fn default() -> DecodePrecision

Returns the “default value” for a type.

impl Eq for DecodePrecision

impl PartialEq for DecodePrecision

fn eq(&self, other: &DecodePrecision) -> bool

Equality operator ==.

fn ne(&self, other: &Rhs) -> bool

Inequality operator !=.

impl StructuralPartialEq for DecodePrecision