skippy-runtime

Safe Rust wrapper around the experimental skippy C ABI.

This crate owns Rust-side model/session wrappers and converts raw ABI buffers into typed runtime structures.

Architecture Role

skippy-runtime is the safe model/session layer used by mesh, servers, diagnostic prompt clients, benchmarks, correctness checks, and slicing tools. It does not own TCP transport, mesh lifecycle, or telemetry export.

flowchart TB
    Mesh["mesh-llm<br/>embedded serving + lifecycle"] --> R["skippy-runtime"]
    P["bench / prompt / correctness<br/>tokenization and local checks"] --> R
    S["skippy-server<br/>prefill/decode requests"] --> R
    ST["HF SafeTensors checkpoint<br/>config + tokenizer + shards"] --> M["skippy-model<br/>mapping + transforms"]
    M --> R
    R --> PS["package selector<br/>manifest + selected parts<br/>direct GGUF fake packages"]
    PS --> R
    R --> F["skippy-ffi"]
    F --> L["third_party/llama.cpp<br/>stage ABI"]

For inference, the runtime opens a stage view, creates a session, runs prefill or decode for that stage's layer range, and returns either an activation frame for downstream stages or a predicted token on the final stage. KV page state export/import also passes through this crate for runtimes that expose native exact-cache movement.

Responsibilities

• open staged model views
• open local Hugging Face SafeTensors checkpoints directly, with optional tensor-by-tensor load-time quantization and no intermediate GGUF
• create runtime sessions
• tokenize and detokenize through llama
• run prefill/decode calls
• set llama.cpp context options needed by staged serving, including K/V cache type selection for upstream-supported cache types and selected backend device placement
• expose activation frames with descriptors and payloads
• select layer-package parts from local or Hugging Face package refs
• validate package ABI/manifest compatibility before loading or composing parts
• expose declared mmproj projector artifacts from package manifests for multimodal serving
• compose materialized package stages for tools that need a concrete GGUF path
• open selected package parts directly through the ABI for server runtime loads

Keep service lifecycle, transport, and telemetry in higher-level crates.

Direct SafeTensors loading

StageModel::open recognizes a local checkpoint directory containing config.json plus either model.safetensors or model.safetensors.index.json. It also accepts a .safetensors file and uses its parent directory for the checkpoint metadata. Set RuntimeConfig::checkpoint_quantization to preserve the canonical checkpoint types or select a supported llama.cpp quantization recipe such as Bf16, Q4KM, IQ2XXS, or Q8_0. Importance-aware recipes use RuntimeConfig::checkpoint_imatrix; relative paths resolve from the checkpoint directory, and required tensor entries are validated before model allocation. The shipped runtime exposes the same settings as mesh-llm serve --quant <RECIPE> and --checkpoint-imatrix <PATH>; explicit CLI values take precedence over both default and model-local configuration.

The runtime validates immutable shards with the official Rust safetensors crate. skippy-model maps and transforms source tensors, while the native Skippy ABI chooses destination tensor types and quantizes each stage-owned tensor as it is loaded. This initial direct path does not materialize a derived artifact cache and does not emit the GGUF model-open progress events.