turbospark-runtime

Raw-completion prefill and decode generation loops (run_raw_completion, run_raw_completion_chunked), LogitProducer trait definition, ScriptedLogitProducer test mock, and RealForwardRunner GPU forward-pass engine (macOS).

Downstream workspace crates import this package via the runtime alias:

[dependencies]
runtime = { package = "turbospark-runtime", path = "../runtime" }

Purpose & Role

turbospark-runtime is the central execution engine of the workspace. It binds tokenization, KV cache management, model forward passes, candidate token selection, and streaming detokenization into a unified generation pipeline. On macOS with Apple Silicon, RealForwardRunner orchestrates GPU compute dispatches for all supported model architectures.

Safety

• #![forbid(unsafe_code)] is enforced in lib.rs.
• Zero raw pointer manipulation or unchecked buffer casts.

Key Modules

• producer.rs: LogitProducer trait definition and ScriptedLogitProducer deterministic test mock.
• raw_completion.rs & raw_completion_chunked.rs: Generation loop drivers orchestrating prefill chunking, step-by-step decode, stop token matching, and streaming callbacks.
• real_forward.rs: RealForwardRunner struct definition, execution lifecycle, and dispatch routing.
• real_forward_open.rs: Model open routines mapping .gturbo weight files and initializing device contexts.
• real_forward_init.rs: Architecture baseline validation, weight buffer binding, and expert streamer setup.
• real_forward_api.rs & real_forward_traits.rs: Public API abstractions for single-token decode and multi-token prefill.
• real_forward_dispatch.rs & real_forward_dispatch_moe.rs: Metal kernel dispatch orchestrators (encode_gemv_any, encode_embed_any, encode_moe_phase1_any, encode_moe_phase2_any).
• real_forward_layout.rs: Quantization dtype translation, block sizes, and MoE blob offset calculations.
• real_forward_rollback.rs: Atomic rollback of KV cache and recurrent state for speculative rejection.
• real_forward_types.rs: RealForwardError, PhaseCounters, dispatch_profile_report, and DecodeScratch.
• real_forward_vision_api.rs & vision/: Vision tower forward pass dispatches and multimodal token injection.
• families/: Dedicated per-architecture forward flows:
  • gemma4/: Gemma 4 attention, MoE routing, and layer norms.
  • gptoss/: Harmony and GPT-OSS architectures.
  • llama/: Llama 3, Mixtral MoE, and related dense models.
  • museglimmer/: Muse Glimmer architecture.
  • qwen/: Qwen 3.5, Qwen 3.6 (GDN hybrid), Qwen 3.8, Qwen 2, and Qwen3-MoE.
  • qwen4/: qwen4_exp QSA indexer and PLE n-gram routing flow.
  • spark/: Spark-X2.5-4B fused QKV and headwise gated architecture.
  • synthetic/: Synthetic fallback flow for headless unit tests.
• steering.rs: Multi-vector composed directional steering applied directly to residual streams during forward passes.
• speculative.rs & speculation_policy.rs: Speculative decoding drivers supporting DFlash block drafting and MTP multi-token prediction heads.
• whisper/: Whisper speech-to-text runner (WhisperRunner), audio mel frontend windowing, encoder dispatches, cross-attention cache, and greedy decode.
• moonshine.rs: Moonshine speech encoder and frontend runner.
• session_pool.rs: Thread-safe session pooling for concurrent or reused model runners.
• turn_stream.rs: Push-callback stream adapter bridging generation loops with async channels and FFI callbacks.
• resid_capture.rs, router_hist.rs, ffn_hist.rs: Diagnostic inspection capturing internal activations, routing weights, and neuron firing histograms.
• kv_prefix.rs & kv_write.rs: Prefix KV cache matching and incremental KV cache writing.
• pacing.rs & power.rs: Decode pacing loops and Apple Silicon thermal/power throttling controls.
• config.rs: Runtime generation configuration and runner parameter tuning.

Development & Test Commands

# Run all unit and integration tests for turbospark-runtime
cargo test -p turbospark-runtime

# Run a specific family integration test
cargo test -p turbospark-runtime --test real_forward_gemma4

Tests

This crate contains 59 integration test suites in tests/:

• Model family forward passes: real_forward_gemma4.rs, real_forward_gptoss.rs, real_forward_llama.rs, real_forward_llama_dense.rs, real_forward_minimax.rs, real_forward_muse.rs, real_forward_qwen.rs, real_forward_qwen2.rs, real_forward_qwen35.rs, real_forward_qwen3moe.rs, real_forward_qwen4.rs, real_forward_spark.rs.
• Chunked prefill & prefix reuse: chunked_prefill.rs, chunked_prefill_refusal.rs, prefix_reuse_real.rs, real_forward_gemma4_chunked.rs, real_forward_qwen35_chunked.rs.
• Quantized KV cache forward tests: real_forward_gemma4_kv_quant.rs, real_forward_gptoss_kv_quant.rs, real_forward_llama_kv_quant.rs, real_forward_qwen35_kv_quant.rs, real_forward_spark_kv_quant.rs.
• Directional steering: real_forward_gemma4_steered.rs, real_forward_gptoss_steered.rs, real_forward_llama_steered.rs, real_forward_museglimmer_steered.rs, real_forward_qwen35_steered_batched.rs.
• Speculative decoding & drafting: real_forward_qwen35_dflash.rs, real_forward_qwen35_mtp.rs, speculative.rs.
• Vision tower integration: vision_tower_parity.rs, vision_tower_synthetic.rs, vision_inject_synthetic.rs, vision_chunked_synthetic.rs, vision_sidecar_synthetic.rs.
• Memory residency & eviction: mapped_expert_residency.rs, mapped_vision_residency.rs.

Crate Gotchas

1. Logit Producer Contract: LogitProducer::produce must return raw, unnormalized logits. selection::select performs softmaxing internally. Returning probabilities destroys sampling temperature scaling (softmax(softmax(z))).
2. Family Selection via ArchConfig: Model execution flow keys strictly on ArchConfig.family, never on weight file names or tensor prefixes.
3. GDN State Reset: Calling reset() on a runner must rewind both the KV cache and GdnStateManager recurrent state (delta rule state S and conv tail buffer) to avoid state bleeding across independent conversation turns.
4. Speculative Rollback Atomicity: In speculative decoding with rejection, rollback must restore the KV cache position, GDN state, and step counter in a single atomic operation before evaluating subsequent draft tokens.