Module native 

Native in-process LLM inference (pure-Rust [candle]) — runs the tuned Qwen3-1.7B + our shipped LoRA fully offline, with no Python serve_openai.py and no HTTP server. This is the ollama-parity path: one binary, models resolved via crate::paths.

Two backends behind one enum:

• Quantized (default, fast, small). A Q4_K GGUF built once from the fp16 base + our LoRA and cached at ~/.steeldb/models/steeldb-qwen3-1.7b-q4k.gguf (~1.1 GB, ~1.1 GB RAM). This is what runs after first setup — comparable to ollama’s footprint.
• Full fp16. Loads the base safetensors and folds the LoRA at load (~3.4 GB). Used only to build the quantized GGUF (a one-time step), or via STEELDB_NATIVE_FP16=1.

LoRA merge folds the PEFT deltas (W' = W + (α/r)·BA) into each targeted projection. Generation speaks the Qwen3 ChatML template (tools rendered into the system turn as <tools> JSON); the returned text is handed to the agent loop, where crate::agent::harness::QwenHarness recovers <tool_call> blocks exactly as for the HTTP Paddock provider.

Structs

GenConfig

Sampling / decoding knobs — mirror train/serve_openai.py (low temperature, repetition penalty).

NativeLlm

NativeProvider

Provider adapter: an in-process NativeLlm behind the async LlmProvider trait. Generation is CPU-bound and &mut, so it runs under a mutex on a blocking thread.

Constants

GGUF_NAME

Cached quantized model filename under the model root.

Functions

build_gguf

Build the cached Q4_K GGUF from the fp16 base + LoRA: merge, map HF tensor names to the llama.cpp convention, quantize matmuls to Q4_K (norms kept F16), and write with the metadata quantized_qwen3 requires. One-time; the result loads fast on every subsequent run.