pub struct Dsv4Layer {Show 24 fields
pub attn_norm: Vec<f32>,
pub ffn_norm: Vec<f32>,
pub wq_a: QTensor,
pub q_norm: Vec<f32>,
pub wq_b: QTensor,
pub wkv: QTensor,
pub kv_norm: Vec<f32>,
pub wo_a: QTensor,
pub wo_b: QTensor,
pub attn_sink: Vec<f32>,
pub compressor: Option<Dsv4Compressor>,
pub indexer: Option<Dsv4Indexer>,
pub hc_attn_fn: Vec<f32>,
pub hc_attn_base: Vec<f32>,
pub hc_attn_scale: [f32; 3],
pub hc_ffn_fn: Vec<f32>,
pub hc_ffn_base: Vec<f32>,
pub hc_ffn_scale: [f32; 3],
pub gate: QTensor,
pub gate_bias: Option<Vec<f32>>,
pub tid2eid: Option<Vec<f32>>,
pub experts: Vec<Dsv4Expert>,
pub shared: Dsv4Expert,
pub mask: Option<Vec<bool>>,
}Expand description
One layer’s weights. Everything quantized rides as QTensor so the
existing kernels (and the mmap) serve them; the small fp32 pieces —
norms, the hyper-connection projections, the sink, the compressor’s
position bias — are plain vectors, exactly as the reference keeps them
in fp32 regardless of the checkpoint’s storage dtype.
Fields§
§attn_norm: Vec<f32>§ffn_norm: Vec<f32>§wq_a: QTensor§q_norm: Vec<f32>§wq_b: QTensor§wkv: QTensor§kv_norm: Vec<f32>§wo_a: QTensor§wo_b: QTensor§attn_sink: Vec<f32>§compressor: Option<Dsv4Compressor>None on the pure sliding-window layers (compress_ratio == 0).
indexer: Option<Dsv4Indexer>Only on the layers whose ratio is 4.
hc_attn_fn: Vec<f32>§hc_attn_base: Vec<f32>§hc_attn_scale: [f32; 3]§hc_ffn_fn: Vec<f32>§hc_ffn_base: Vec<f32>§hc_ffn_scale: [f32; 3]§gate: QTensor§gate_bias: Option<Vec<f32>>noaux_tc selection bias — None on the hash layers.
tid2eid: Option<Vec<f32>>Token-id → expert table on the hash layers, None elsewhere.
experts: Vec<Dsv4Expert>§mask: Option<Vec<bool>>Task-conditional restriction over the routed experts
(CMF_MOE_MASK + CMF_MOE_MASK_COVER): false experts are not
selectable and the weights renormalize over what remains. None on
the hash layers — their table names specific experts, so masking
there would silently reroute rather than restrict.