hf2q 0.1.1

Pure Rust CLI for converting HuggingFace models to hardware-optimized formats and serving them over an OpenAI-compatible API on Apple Silicon
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37

//! Per-architecture Gemma 4 inference support.
//!
//! Owned by ADR-008 (mlx-native forward path) and ADR-017 (TQ-packed KV
//! persistence). Variant detection in `serve::api::engine::LoadedModel::load`.
//!
//! Migration history: ADR-038 split this from monolithic `src/serve/forward_mlx.rs`.
//!
//! # Module layout
//!
//! * `model.rs` — `MlxModelWeights` struct, GGUF loader, DWQ overlay, embed_tokens.
//! * `kv_cache.rs` — KV-buffer structs + `DecodeRegime` (Step 2).
//! * `gpu_full_attn.rs` — per-layer attention dispatch (`encode_one_layer`).
//! * `gpu_ffn.rs` — Path A stub; Path B follow-up will extract from `encode_one_layer`.
//! * `forward_gpu.rs` — outer `forward_decode` + ADR-031 parallel worker.
//! * `io_heads.rs` — argmax/logits/NLL surface.
//! * `kv_persist.rs` — ADR-017 TQ snapshot/restore.
//! * `profile.rs` — kernel/token profile + accumulator.
//! * `tokenizer.rs` — GGUF-embedded tokenizer (unchanged).

pub mod batched_body;
pub mod batched_head;
pub mod forward_gpu;
pub mod gpu_ffn;
pub mod gpu_full_attn;
pub mod io_heads;
pub mod kv_cache;
pub mod kv_persist;
pub mod model;
pub mod profile;
pub mod tokenizer;

// Re-exports collapse import-site churn for the most-touched types.
pub use kv_cache::{
    DecodeRegime, DenseKvBuffers, GemmaLcpLayerKv, HbKvBuffers, HybridKvBuffers, MlxKvCache,
};
pub use model::MlxModelWeights;
pub use profile::{KernelTypeProfile, ProfileAccumulator, TokenProfile};