Struct NativeSafetensorsLoader 

pub struct NativeSafetensorsLoader<B: Backend + BackendQuantMarlin> { /* private fields */ }

Native safetensors loader. Generic over Backend so every tensor is materialised directly into backend-native buffers.

Implementations

impl<B: Backend + BackendQuantMarlin> NativeSafetensorsLoader<B>

pub fn open(model_dir: impl AsRef<Path>) -> Result<Self>

Discover shards under model_dir and build the name → shard index.

pub fn read_gptq_raw( &self, name: &str, ) -> Result<(Vec<i32>, Vec<f32>, Vec<i32>, Option<Vec<i32>>, usize, usize)>

Read the four raw GPTQ tensors for a named projection without triggering a Backend repack. Used by MoE batch loading: callers stack many experts host-side then issue a single B::load_gptq, avoiding the 12 288× per-expert Marlin repack overhead.

Returns (qweight, scales, qzeros, g_idx, k, n). g_idx is None when desc_act=false (no act-order perm needed).

pub fn quant_config_ref(&self) -> Option<&QuantConfig>

pub fn load_stacked_gptq_experts( &self, expert_prefix_fmt: &str, num_experts: usize, proj_names: &[&str], ) -> Result<(Arc<dyn MarlinExpertStack<B>>, usize, usize)>

Load a STACKED GPTQ tile that concatenates num_experts experts’ raw GPTQ tensors along the N (column) axis and runs ONE backend repack — instead of num_experts × proj_names.len() repacks.

Layout: per row r, the cols are emitted in expert-major order: expert_0[proj_0|proj_1|...] | expert_1[...] | ... | expert_{N-1}[...]. Caller can therefore index expert e at column offset e * n_per_expert, where n_per_expert = Σ n(proj) across the proj_names for one expert.

expert_prefix_fmt should be a closure-style &str that contains "{e}" placeholder (replaced by the expert index) and ends just before the proj name — e.g. "model.layers.5.mlp.experts.{e}.". The full tensor name probed is {expert_prefix}{proj}.

Returns (store, n_per_expert, k) where n_per_expert is the per-expert column width and k = in_features (shared by all).

Trait Implementations

impl<B: Backend + BackendQuantMarlin> WeightLoader<B> for NativeSafetensorsLoader<B>

fn load_tensor(&self, name: &str) -> Result<B::Buffer>

Load a single tensor by fully qualified name (e.g. "model.embed_tokens.weight").

fn load_linear(&self, name: &str) -> Result<Box<dyn Linear<B>>>

Load a projection as a Linear<B>. The concrete implementation (DenseLinear / GptqLinear / AwqLinear / GgufLinear) depends on the loader’s file format and quant config.

fn has_tensor(&self, name: &str) -> bool

Whether a tensor with this name exists in the source.

fn quant_config(&self) -> Option<&QuantConfig>

Quantization metadata (parsed from quantize_config.json or a GGUF header). None means the source is dense.